Background <p>Organoid image segmentation is essential for quantitative analysis in disease modeling and drug screening, yet remains highly challenging due to substantial morphological variability and blurred boundaries in organoid images. Existing approaches often struggle to achieve a favorable balance between segmentation accuracy and computational efficiency.</p> Results <p>In this paper, DeShiftNet, a lightweight segmentation framework, is proposed to extract discriminative features with high accuracy while maintaining low computational overhead. The model incorporates a deformable-shifted encoding strategy that adaptively samples local structures. It also includes a cross-attention-guided decoder for selective multi-scale feature alignment. Furthermore, a deformable multi-scale contextual refinement module enhances boundary coherence and contextual consistency. Extensive experiments on the multi-type OrganoID dataset show that DeShiftNet achieves competitive performance compared with recent segmentation models, while maintaining only 1.78M parameters and 2.65 GFLOPs. Notably, DeShiftNet achieves a Dice score of 0.961 on the Lung subset.</p> Conclusion <p>These results indicate its potential practical value for efficient organoid segmentation in high-throughput experimental workflows.</p>