Ultrasound image segmentation plays a critical role in medical-assisted diagnosis but suffers from inherent limitations, such as high noise, artifacts, and morphological diversity. Existing methods struggle to generalize with small-sample data due to feature contradictions from varying acquisition angles, limiting multi-center clinical use. To address these issues, we propose a dual prior-guided two-stage segmentation framework. In the first stage, the prior classification of small-sample data guides domain adaptation pretraining on large-scale datasets, employing dynamic class balancing to mitigate data distribution bias. The second stage features a multi-level feature fusion architecture with three core modules: First, we design a Multi-branch Convolutional Parallel Attention (MCPA) module that extracts contextual features via parallel dual attention to adaptively select multi-scale features. Next, we propose a Multi-scale Fusion Dilated Convolution (MFDC) module that enhances the encoder’s capability to capture lesion boundaries across different receptive fields through hierarchical dilated convolutions. Finally, we introduce an Enhanced Feature Decoding module (EFD) in the decoder, embedding a cross-layer compensation mechanism using shallow high-resolution features to recover spatial details lost. Furthermore, we propose an interactive dual-stream architecture that bridges prior-guided classification and segmentation tasks, where complementary features are fused through cross-task attention to optimize holistic semantic consistency and robustness. Experiments on the public dataset demonstrate our method’s superiority over mainstream approaches. Ablation studies validate the effectiveness of our method, providing a solution for high-precision, high-availability small-sample ultrasound image segmentation. Code is on Github: https://github.com/notchXie/DPGS-Net.