Fabric defect detection algorithms currently suffer from low detection accuracy, slow processing speeds, and high rates of missing subtle defects, severely limiting the application of this technology. To address this challenge, this paper proposes a fabric defect detection algorithm called GDC-YOLO, which contributes as follows: (1) To reduce network parameters and computational costs, we utilize the Ghost module to enhance the detection capability of tiny defects while lowering model complexity. (2) We introduce a novel DWAM attention mechanism, leveraging depthwise separable convolution, to resolve the issue of spatial and channel attention interfering with each other. (3) To better understand contextual information in feature maps and spatial relationships, we incorporate coordinate information into the feature maps, thereby enhancing the model’s generalization ability. (4) Addressing the issue of long-tailed data, we effectively mitigate the impact of imbalanced positive and negative samples on detection results using Focal Loss, thereby improving the accuracy and stability of fabric defect detection. On the SDCF dataset, compared to YOLOv5, GDC-YOLO significantly reduces network parameters by 20.4%, while simultaneously improving , accuracy, and recall by 5.5%, 6%, and 5.2%, respectively. Additionally, the proposed DWAM attention mechanism serves as a superior alternative to CBAM, significantly increasing by 1.7% in experiments.