To enhance the accuracy, real-time performance, and segmentation capabilities of deep learning perception models in rail transit autonomous driving scenarios, this study proposes a hybrid instance segmentation model (Dfine-Mask) by fusing the D-Fine detector and Mask-DINO segmentation module. First, the model retains D-Fine’s core mechanisms—Fine-grained Distribution Refinement (FDR) for bounding box regression optimization and Global Optimal Localization Self-Distillation (GO-LSD) for knowledge transfer—to ensure high-precision target localization. Second, it integrates Mask-DINO’s mask-aware queries and denoising training module into D-Fine’s decoder, enabling pixel-level instance segmentation while maintaining efficiency. Experiments were conducted on a self-constructed rail transit dataset (20,000 training images +3,000 test images, covering 5 categories, pedestrians, trains, obstacles, buffer stops, and signal lights). Results show that the Dfine-Mask model achieves 0.970 mAP@0.5 and 0.762 mAP@0.5:0.95 on the test set, and its inference latency is 68 ms per frame on the NVIDIA-Xavier platform (faster than most YOLO series models) and its average mask IoU reaches 0.82, effectively supporting advanced perception tasks such as track intrusion detection and obstacle contour extraction. This model provides a reliable technical solution for environmental perception in rail transit autonomous driving.