Medical image segmentation aims to delineate anatomical structures across various imaging modalities, such as CT and MRI. The Segment Anything Model (SAM) has been widely applied in medical image segmentation. However, challenges arise when transferring it to the medical domain, particularly for 3D image segmentation, due to inherent differences in imaging modalities, spatial configurations, and semantic contexts. To bridge this gap, we propose DPA-SAM, a novel SAM adaptation framework tailored for medical imaging. DPA-SAM introduces two effective modules without altering the frozen SAM encoder: (1) 3D deformable channel attention fusion combines volumetric deformable convolutions with channel attention; (2) parameter-guided attention incorporates learnable priors. These modules are integrated into the image encoder and mask decoder respectively, enabling efficient and targeted enhancement of spatial adaptability and local anatomical perception, while simultaneously refining boundary awareness and structural representation. We evaluate DPA-SAM on three representative medical segmentation tasks. Experimental results demonstrate that DPA-SAM consistently outperforms existing 3D segmentation methods, achieving an average Dice score of 89.7%(improved by 2.5% compared with state-of-the-art method) on the abdominal multi-organ segmentation dataset, with particularly notable gains in Dice scores for small or low-contrast structures.