Over the past few decades, many algorithms for object detection (OD) in natural scene images have been proposed. However, due to the characteristics of optical remote sensing (RS) images—such as high resolution, complex backgrounds, large variations in object scale, and dense distributions of small targets, directly applying these detectors to optical RS images is often ineffective. To address these challenges, this paper introduces YOLO-PFS, an efficient detector based on YOLOv11n. First, we propose a new feature extraction module, PFS, which better handles features at multi-scales in the image, helping the network understand and learn the input data more comprehensively. Next, The addition of a convolution and self-attention fusion module (CAFM) to our network results in the SPPFC module, which enhances the feature extraction process and significantly improves the network’s detection performance for small objects. Additionally, we add two extra detection heads to further enhance detection performance. we introduce Inner-WIoU loss function to reduce the impact of scale differences on detection accuracy. On the optical remote sensing dataset SIMD, YOLO-PFS achieved an AP50 of 84.4%, with an average precision improvement of 3.0% compared to the baseline model. With a model size of n and no pre-trained weights used, YOLO-PFS becomes the state-of-the-art model, 1.7% higher than the AP50 of YOLO-HR-n. Further validation on the DIOR dataset demonstrated the superiority of YOLO-PFS, with AP50 and mAP reaching 88.1% and 65.6%, respectively, outperforming the baseline model by 1.7% and 10.7%, achieving better detection performance.