The Segment Anything Model (SAM) has emerged as a promising tool for image segmentation in the age of large-scale models. Despite being trained on an extensive dataset of over 11 million natural images, the nuanced and critical nature of pathological images challenges SAM’s applicability. Leveraging an expansive dataset of 28,786 patches across five modalities, we introduce a two-stage segmentation strategy tailored for SAM. The initial stage deploys convolutional neural network-based density regression models for cell detection, while the subsequent stage uses these detected regions as prompts to guide SAM’s segmentation task. Our model has achieved outstanding performance on both stages and provides a general paradigm for the detection and segmentation of pathological cells. Besides, this paper offers an extensive evaluation of SAM’s performance in pathological image segmentation, with a specific focus on multi-stained cell segmentation. Our results indicate that SAM’s performance varies across distinct pathological modalities when guided by bounding box prompts, which proved the need of developing a more robust model for pathological image segmentation in the future.