The rapid evolution of unmanned aerial vehicle (UAV) technology poses substantial obstacles to the effectiveness of anti-UAV detection methodologies. Addressing the challenges associated with the small UAV target size, indistinct features, and detection difficulties in complex infrared images, this paper proposes an IR anti-UAV detection algorithm named RFW-YOLO. Initially, considering small target sizes in IR images, we design an RFPN to improve the Neck of YOLOv11. The proposed SBA module incorporates a bidirectional fusion mechanism to adaptively integrate shallow feature boundary details with deep feature semantic information, thereby delineating finer object contours and recalibrating object locations. Secondly, to further enhance feature extraction capabilities, we integrate a WTConv module into the YOLOv11 backbone network. Leveraging the multi-resolution analysis properties of wavelet transforms, this module expands the receptive field and extracts richer features without significantly increasing parameters. Furthermore, to mitigate the issue of insufficient sample data in anti-UAV IR datasets, we employ transfer learning, initializing the model with pre-trained weights derived from a large-scale IR dataset to accelerate model convergence and boost detection accuracy. Finally, to validate the algorithm's effectiveness, we conducted experiments on an anti-UAV IR dataset. The results demonstrate that, in comparison to the YOLOv11n, our approach improves Precision, Recall, mAP, and mAP50-95 metrics by 2.9%, 5.7%, 3.8%, and 4.2%, respectively. This underscores the efficacy of our algorithm in improving the detection performance of IR anti-UAV targets.