Recent trends indicate a growing rate of course failures among students in educational institutions. Consequently, predicting students’ learning outcomes (LO) has become crucial in providing timely warnings for students and assisting educators in adapting teaching plans to enhance training quality. A key sub-problem in learning outcome prediction is the classification of students’ learning outcomes. However, this task is often hindered by the imbalance in the dataset, as the number of students failing to meet requirements is significantly smaller than those who pass. This imbalance leads to a notable reduction in the prediction performance for students who do not meet the criteria. This paper addresses this issue by proposing the SMOTERN method, which aims to improve the prediction efficiency for students with unsatisfactory learning outcomes. The method involves removing noise from minority classes based on the number of neighbors before generating synthetic samples using the SMOTE algorithm. Using the OULAD dataset, we classify learners into pass and fail categories, where the number of pass learners is approximately half that of fail learners. Experimental results demonstrate that the proposed SMOTERN method significantly enhances classification performance. Models trained on data adjusted using SMOTERN show an improvement in AUC scores by 4–10% and F2 scores by 2–10% compared to other methods.