In agricultural production, biological pest detection relies on tools such as yellow sticky traps to capture pests and then detect them. The detection of pests is crucial to ensuring the health and yield of crops. However, computer vision-based research on small pest detection confronts challenges related to image quality, including issues caused by lighting changes and the struggle to retain detailed features of small pests. To address the problem, a pest detection network called RGPest-YOLO is proposed, which is based on image preprocessing techniques. The proposed method achieves a mAP50 of 85%, a precision of 85.4%, and a recall of 79% on the synthetic pest dataset Bdata. An average precision (mAP50) of 73.2% is achieved on the real dataset ZPest, marking a 3.8 percentage point improvement over the original YOLOv8 model. When compared to current mainstream object detection methods, this approach demonstrates superior performance. The performance of small pest detection is significantly enhanced, and robust technical support is provided for pest monitoring within smart farming systems.