Kidney stone segmentation plays a critically important role in the management of kidney disease, feature extraction, disease localization, and identification. However, existing methods often face issues such as unclear boundaries and blurred details of kidney stones, resulting in suboptimal segmentation performance. In this paper, we describe the depth-separable convolutional for the first time and propose a novel Channel and Pyramid Dual Branch Attention Network (CPDBA-Net) to resolve these issues. In particular, we introduce a newly Channel and Pyramid Dual Branch Module (CPDBM), which combines the local attention mechanism of Convolutional Block Attention Module (CBAM), and the multi-scale feature extraction of Atrous Spatial Pyramid Pooling (ASPP). The module enhances the model with the following capabilities to focus on kidney stone details and background, overcoming the limitations of traditional convolutional neural networks in handling fine details and multi-scale target segmentation. The experimental findings indicated that CPDBA-Net is an effective kidney stone segmentation model and outperforms the current state-of-the-art models.