<p>Remote sensing images (RSIs) have become integral to a multitude of sectors, including military operations, urban traffic planning, and natural resource management, making the detection of targets within RSIs a critical research endeavor. Although RSIs object detection holds significant application value across various domains, its research still confronts numerous challenges. These include the interference of complex backgrounds in RSIs, the high resolution of RSIs that complicates deployment on computationally constrained satellite platforms, and processing large amounts of complex data leads to inefficiencies. To surmount these obstacles, we introduce a novel Task-Balanced Algorithm for Object Detection in Remote Sensing Images Based on Improved YOLO—MTGS-Yolo. The algorithm commences with the construction of a Multi-Transformer model, designed to address dense prediction problems. By significantly augmenting the network's capacity to capture both local and global contextual information, it minimizes information loss and enhances the network’s adaptability to more intricate background scenarios. Furthermore, we have incorporated a Generalized Efficient Aggregation Network (GELAN) structure, which transcends traditional architectural and device limitations. This innovation is geared toward adapting to the spectrum of feature learning, from complex to lightweight, thereby achieving a model that is both lightweight and computationally efficient. This advancement not only reduces computational costs but also significantly improves model efficiency. In response to the issue of low feature resolution for small objects in RSIs, which often leads to background confusion, we have proposed a Spatial Context-Aware Module (SCAM). This module leverages spatial contextual information to delineate cross-spatial relationships between pixels, effectively suppressing irrelevant background elements and enhancing the distinguishability between targets and their surroundings. Experimental results on the public DIOR dataset demonstrate that MTGS-Yolo surpasses the baseline network in terms of detection performance and robustness. Additionally, transfer learning experiments conducted on the NWPU VHR-10 dataset reveal that MTGS-Yolo outperforms other classic and improved algorithms in terms of detection performance and exhibits superior generalization capabilities.</p>