Riveting is the main connection method in aircraft assembly, and its detection is crucial to ensure flight safety. In view of the problem that traditional manual visual inspection methods are difficult to meet the needs of large-scale and high-precision detection, a fast rivet detection algorithm based on the VanillaNet network is proposed. First, by using the VanillaNet network architecture, the complex backbone part in the traditional YOLO series algorithm is replaced, effectively reducing the complexity of the model. Secondly, the Bi-FPN (Bidirectional Feature Pyramid Network) and the RFA (Receptive-Field Attention) detection head combined with the spatial attention mechanism are introduced to improve the multi-scale detection performance and difficult sample feature extraction capabilities of the model respectively. In addition, an industrial-grade dataset containing a variety of rivet models and defect classifications is constructed, and the generalization ability of the model is improved through a variety of data enhancement techniques. Experimental results show that compared with the YOLOv8 baseline model, the fast rivet detection algorithm has a 6.7% increase in average accuracy and a 126% increase in detection speed. The fast rivet recognition algorithm provides a new solution for automated detection in the aviation manufacturing industry and has broad application prospects.