<p>The grape stems exhibit significant visual similarity to petioles and demonstrate low-pixel-coverage characteristics in visual imagery. To accurately identify grape stems, we propose the “Find the Stem Along Grape” method, which locates the stem. First, the positions of all grape bunches are detected within the complex panoramic image. Then, the stem of individual bunch is identified to minimize interference from petioles, which significantly improve the accuracy of instance segmentation for both grapes and stems. Next, a region-based image segmentation strategy is proposed, in where an instance segmentation model is trained on images containing only grape bunches. It allows the model to focus on the fine-grained features of grapes and stems, while effectively reducing background noise and improving segmentation performance on panoramic images. Finally, Residual Network 50 with Feature Pyramid Network (ResNet50-FPN) is used as the backbone network of Mask Region Convolutional Neural Network (Mask R-CNN). To improve Mask R-CNN’s perception of fine-grained stem features and spatial information while enhancing multi-scale feature representation, we incorporate Efficient Convolutional Block (ECB) and Dense Upsampling Convolution (DUC). The results for using the region-based labeling method shows that the segmentation accuracy of the augmented model for grapes and stems reached 98.7% and 90.1%, respectively. It is 1.5% and 6.1% higher than that of the panoramic labeling method. The method enhances the accuracy of stem segmentation in the complex background of vineyards and contributes to intelligent grape harvesting.</p>