Adaptive learning systems are revolutionizing education, particularly in mathematics, by offering personalized learning experiences that cater to individual students. This research classifies students based on their behavior and applies machine learning algorithms to determine the relationship between student’s behavior and academic achievement. The student clustering method identified three groups of students: high achievers, average achievers, and underachievers. Several predictive models like XGBoost, Random Forest, Logistic Regression, Decision Tree, K-Nearest Neighbors, and Multi-Layer Perceptron were applied to make predictions on final scores using first test scores and cluster labels. These findings indicate how crucial it is to group behavior in adaptive learning and how machine learning assists in prediction and improvement of the learning system. They also indicate that we must incorporate more affective data, such as facial emotion analysis, to improve predictions. These findings assist in offering better personalized educational assistance by providing insightful information on students’ learning behavior.