High-accuracy and low-latency are critical for surface defect detection in industrial quality control. However, existing YOLO-based methods frequently struggle to balance detection performance and computational efficiency, particularly on resource-constrained edge devices. To address these challenges, this paper proposes LA-YOLO, a new lightweight architecture designed to optimize both efficiency and efficacy by improving the backbone, neck, and detection head. Specifically, to enhance backbone efficiency, we present a series connection strategy utilizing Mobile Inverted Bottleneck Convolution across shallow and deep layers. Additionally, we reduce computational overhead by integrating partial convolution with standard convolution operations. Furthermore, to improve small defect detection, we propose a dual-head adaptive spatial fusion mechanism in detection head. Experiments results on two industrial defect databases demonstrate that LA-YOLO achieves a 2–5x reduction in FLOPs while significantly improving inference speed compared to the state-of-the-art methods. Moreover, comparing with the state-of-the-art YOLO variants, the results highlight LA-YOLO’s superior performance and its suitability for real-time industrial inspection systems.