Coffee bean quality directly determines the coffee quality, therefore accurate and timely detection of bean defects is crucial for coffee bean roasting. The development of deep learning effectively addresses the challenges of high labor requirements and subjectivity of traditional manual screening methods. However, existing research has limited the development of coffee defect recognition techniques by detecting few categories and mixing overall and localized defects. To address the above problems, we created a coffee bean defects dataset with five types and proposed a novel two-step defect recognition model (Res-FMYOLO) which comprises Primary Classification and Reclassification steps. In primary classification step, ResNet 34 was used to identify different overall defects, achieving a remarkable Accuracy of 98.42%. In reclassification step, we propose an efficient and lightweight coffee bean defect detection model FMYOLO to accurately localize local defects and classify them precisely. Specifically, we designed a Feature Focus Bottleneck (FFB) by combining the standard Bottleneck with Efficient Channel Attention to enhance the perception ability of the model on the color feature changes caused by defects. Second, we embed the Multi-scale Cross-axis Attention in Neck to efficiently establish long-range dependency between pixels to mitigate the problem of large variation in defect size and shape. FMYOLO achieves a remarkable mean Average Precision of 96.20%, outperforming mainstream object detection networks. This study reveals the impact of mixing overall and local defects on defect classification accuracy, and the proposed model promises efficient and accurate coffee bean defect detection, providing technical support for defect detection in coffee.