China’s tea industry experiences substantial demand, necessitating advanced pest and disease detection technologies for effective crop management. Tea cultivation is frequently challenged by insect infestations and brown spot disease; yet, detecting these issues in tea leaves, especially in natural environments, is complicated by complex backgrounds and small target regions. Addressing these challenges requires refining the you only look once (YOLO) model with techniques optimized for small target detection, utilizing deep learning approaches. To enhance detection accuracy, data augmentation methods, including rotation, flipping, and contrast adjustment, were applied to expand the dataset and improve model robustness. This study introduces the optimized weight network (OWN) model, an enhanced version of the YOLOv11 architecture, incorporating a P2 detection layer and the inverted residual block (iRMB) attention mechanism. In addition, the model integrates the cross-stage partial spatial attention (C2PSA)_iRMB module and P2/4-xsmall layers. The P2 detection layer strengthens image feature extraction, improving the identification of small targets, while the iRMB attention mechanism enhances feature selection efficiency, minimizing redundancy and optimizing model performance. Experimental results indicate that the OWN model achieves a mean average precision (mAP@0.5) of 90.8%, a precision of 93.1%, and a recall rate of 97%. Furthermore, the model maintains a stable size, effectively balancing efficiency and performance. These findings highlight the model’s effectiveness in detecting tea pests and diseases under complex environmental conditions.