<p>Fabric defect detection is critical for ensuring quality assurance in textile manufacturing. However, fabric defect often has low contrast and diverse morphologies, resulting in poor detection performance for existing lightweight target detection algorithms. To address these challenges, an efficient gradient aggregation lightweight network (EGAL-Net) is designed in this study. First, an efficient gradient aggregation module (EGAM) is designed, which can perform efficient feature extraction while meeting the lightweighting requirements. Second, a feature-guided fusion pyramid network (FGFPN) is proposed, which consists of two core modules: the dynamic feature selection (DFS) module and the bidirectional interaction and dynamic fusion (BIDF) module, which work together to achieve efficient cross-layer aggregation and ability to discriminate between key features. In addition, the selective downsampling (SDown) was proposed, which integrates standard and linear deformable convolutions into a branched parallel structure to adaptively adjust the sampling positions for different object shapes. The proposed model is experimentally validated on the self-made ZhuoYuan dataset and the publicly available DAGM2007 and TianChi datasets. On the ZhuoYuan dataset, the mAP50 of EGAL-Net improves by 6.3%, the parameters are reduced by 45.7%, and the GFLOPs are reduced by 27.8% compared to the original model, significantly enhancing performance while maintaining efficiency. Results on the public datasets further confirm the robustness and generalization of EGAL-Net.</p>