<p>The balling phenomenon is a prevalent defect in laser powder bed fusion (LPBF), adversely affecting the fatigue life and mechanical properties of final parts. Therefore, reasonably controlling the generation of balling defects during the forming process is crucial for maintaining process stability and achieving high-quality parts. Fortunately, balling defects are formed with the aggregation of metal pellets and only appear on the interlayer surface of the as-built part. This characteristic makes it possible for layer-wise monitoring and control. In response to this challenge, this study presents an optimized convolutional neural network (CNN) for the robust recognition of balling levels, which indicate the severity of balling defects and thus help predict part quality. Specifically, a basic CNN model with a specific architecture is explored based on a balling image dataset collected from practical printing processes. To enhance the model’s performance, a non-linear genetic algorithm (NGA) is proposed to find optimal regularization coefficients, which have been verified to alleviate the overfitting present in the basic CNN model. This optimized model, referred to as NGA-CNN, achieves a recognition accuracy of 96.27% on a small segmented image dataset, surpassing both the basic CNN model and those CNNs optimized using other artificial swarm intelligence algorithms. Moreover, adjustments to process parameters, informed by the recognition results, effectively reduce the occurrence of the balling phenomenon. This study will provide an effective method to realize real-time monitoring and control of forming quality in the LPBF process.</p>