<p>Stainless steel pipes are one of the indispensable raw materials in industrial production, and they are widely used in fields such as aerospace, energy, and chemical industries. The service environment involves harsh conditions such as high temperature, high pressure, corrosion, and fatigue, placing high demands on structural safety. Due to the complex surface defects of steel pipes and the similarity of defect types, the assessment of the surface quality of steel pipes still relies on manual visual inspection. Based on these issues, this paper proposes a semi-bilateral efficient self-attention network (SBSANet) for quantifying the severity of internal surface defects in steel pipes. Firstly, a semi-bilateral and jump connection structure was designed to effectively compensate for information loss during the encoding process. Then, a multi-head attention encoding (MHAE) module was proposed to enable long-distance pixel interactions. Finally, a multi-scale residual context extraction (MSRCE) module was designed at the end of the encoder, effectively extracting multi-scale contextual information and enhancing the network’s segmentation capability. Experimental results show that compared to other networks, the proposed method exhibits superior performance. SBSANet achieves a mean intersection over union (mIoU) of 78.48% on a self-made dataset and a running speed of 91.62 FPS, with mIoU of 81.28% and a running speed of 92.70 FPS on the NEU-Seg strip steel dataset.</p>