Detecting targets in remote sensing imagery is essential for various real-world applications. However, single-modal approaches often underperform when faced with small-scale objects, complex backgrounds, and inconsistent imaging environments. To address these limitations, we propose PMF-YOLO, a cross-modal system tailored to boost the recognition of small-scale objects in remote sensing data. PMF-YOLO introduces a Dual Scale Cross-Fusion (DSCF) module that utilizes a dual-layer dilated convolution mechanism to extract comprehensive contextual features from thermal and visible-spectrum images. This module strengthens small target detection by leveraging enriched contextual information. Following the cross-fusion of features from both modalities, a spatial selection is applied to optimize feature discrimination over different scales. The Local Region Attention (LRA) module further optimizes feature aggregation by concentrating on crucial local regions within the input image. It employs an adaptive weighting strategy grounded in Euclidean distance to highlight essential features associated with the target, thereby enhancing the recognition of tiny or visually indistinct targets. Additionally, the Multiscale Adaptive Fusion (MAF) module mitigates inconsistencies among multiscale feature maps by dynamically integrating spatial information across various scales. This module preserves computational efficiency while reinforcing feature representation and ensuring scale invariance. Evaluation on the VEDAI and DIOR datasets confirms that PMF-YOLO surpasses existing methods in detection accuracy and efficiency, offering a robust solution for small target identification in intricate remote sensing environments.