The detection of surface defects in industrial settings has become a research focus within the area of computer vision. It is a difficult and demanding task because the defects are difficult to distinguish due to complex background interference. In this article, we present a deep learning framework that leverages multi-scale feature fusion, which is specifically used for detection of defects on metal surfaces. We focus on the information communication and transfer capabilities of the model to upgrade the accuracy of surface defect detection. First, we introduce the spatial attention channel module (SACM), which can effectively integrate multi-scale feature information. Through parallel independent calculations, each processing unit can focus on information at different scales, thereby providing richer details and contextual support for subsequent defect detection. Secondly, to address the issue that traditional methods can’t effectively filter out background interference during the decoder information recovery process, we propose a more effective upsampling module CAFUEA. This module specifically enhances defect features to suppress background noise and highlight defect information. In addition, in an effort to capture multi-scale information more effectively, we also propose a new global complementary information integration module (GCII). This module further enhances the stability of the model to complex backgrounds by integrating global information. Experimental results on NEU-Seg and USB-Seg datasets indicate that our model outperforms other popular methods in segmentation accuracy.