Traditional methods for identifying walnut appearance defects during initial processing suffer from misdetection, missed detection, and are time-consuming and labor-intensive. In addressing these issues, we propose a proven detection model named ESA-YOLO11(EIER-SlimNeck-ADown, ESA), based on YOLO11(You Only Look Once version 11) architecture. Firstly, the EIER module is presented and fused with some of the C3k2 modules in the backbone. This module combines the SobelConv branch for extracting edge information and the convolution branch for extracting spatial information, allowing for the learning of more abundant image feature representations. Secondly, the Slim-Neck structure is integrated in the neck, reducing the network complexity while ensuring the effective recognition capability of walnut appearance defects. Additionally, the ADown downsampling module is introduced in the backbone to replace the standard convolution module, reducing the model’s parameter count and computational cost and optimizing the feature extraction process. The results show that compared with the original model, ESA-YOLO11 has increased mAP 50 by 3.1%, and reduced the number of parameters and computational cost by 15% and 18% respectively. This model achieves precise identification of walnut appearance defects and can provide support for the automated detection of walnut sorting.