<p>Coal is a key resource in the global energy supply. Existing RT-DETR- and YOLO-based detectors suffer from limited feature representation, poor robustness in complex underground environments, and high computational cost in achieving a balance between accuracy and efficiency. To mitigate potential safety hazards and reduce accidents in complex underground coal mines, this paper proposes GSSE-DETR, an RT-DETR model for real-time detection of personnel behavior. The SENetV2 backbone network is integrated to strengthen the channel attention mechanism, thereby extracting higher-quality and more information-rich initial features. It introduces a novel feature fusion module, GVFF. By replacing conventional convolutions with GSConv modules, this design improves cross-scale feature fusion while enhancing overall efficiency and real-time performance. Furthermore, the VoV-GSCSP module is used to further reduce computational complexity. Experimental validation on the DsLMF+ dataset indicates that the proposed GSSE-DETR model achieves 84.44% precision, 81.21% recall, and 78.21% mean average precision. These metrics represent improvements of 1.94%, 10.65%, and 4.72%, respectively, over the original RT-DETR model. Compared to mainstream models, such as YOLOv10n and YOLOv11n, GSSE-DETR demonstrates superior stability and generalization. Consequently, the targeted combination and structural optimization of multiple mature modules enable the proposed method to achieve an optimal balance between accuracy, model size and computational efficiency. This customized improvement scheme for underground coal mining scenarios provides a practical, robust solution for on-site, real-time monitoring of personnel behavior.</p>