<p>Remote sensing object detection plays a crucial role in various fields, including urban transportation, agriculture, and logistics. However, challenges such as limited object information and complex backgrounds often hinder achieving a balance among performance, efficiency, and lightweight design. To address this issue, we propose ERS-DETR, a real-time remote sensing object detection transformer model. First, a lightweight backbone network, ELCGNet, is designed to enhance the ability of extracting global and local features while reducing the number of parameters by 25%. In addition, ERS-DETR incorporates a high–low-frequency dual-channel encoding derived from HiLo attention, optimizing the detection of small objects by better extracting multi-frequency features. Second, ERS-DETR introduce ERSO-FPN, a feature fusion pyramid that integrates large kernel convolutions and a dual-domain attention mechanism, thereby improving the effectiveness of global-to-local feature learning. Lastly, the Focaler-MPDIoU loss function is adopted to better handle class imbalance, enabling the model to focus on challenging remote sensing targets. Experimental results on the VisDrone-2019, RSOD, and TinyPerson datasets demonstrate that ERS-DETR achieves mAP@0.5 scores of 49.0%, 94.9%, and 21.8%, respectively, representing improvements of 3.3%, 1.6%, and 2.2% over the baseline model. These results highlight that ERS-DETR improves detection performance in complex environments while maintaining a lightweight and efficient structure. The source code is available at <a href="https://github.com/LzmSneak/ERS-DETR">https://github.com/LzmSneak/ERS-DETR</a></p>