Detecting small targets in remote sensing imagery is frequently impeded by target faintness and complex background, resulting in reduced accuracy. This work introduces an Enhanced Small Target Detection method, termed ESM-YOLO, which leverages multi-modal fusion of visible and infrared data to enhance inter-modality correlation and thereby augments performance. Firstly, we devise a pixel-level Bilateral Excitation Fusion (BEF) module to extract both shared and unique features from distinct modalities symmetrically and efficiently. Subsequently, an Improved Atrous Spatial Pyramid Pooling (IASPP) unit and a Compact BottleneckCSP (CBCSP) unit are incorporated into the detection architecture. These components are meticulously tailored to enhance the detection of minute object features, while ensuring a balance between computational efficiency and feature representation capability. Experimental results show that ESM-YOLO achieves 82.42% accuracy on the widely used Vehicle Detection in Aerial Imagery (VEDAI) dataset. The effectiveness and superiority of our proposed method are demonstrated through extensive experiments.