Validating Generative Educational AI: A Framework for Expert-Evaluation of LLM-Generated Personalized Learning Recommendations
摘要
Adaptive learning platforms generate extensive student data, enabling the prediction of academic risks, but often fail to provide students with actionable, personalized guidance for improvement. This paper introduces a novel framework that leverages machine learning for dropout prediction and Large Language Models (LLMs) for generating natural language, personalized learning recommendations. Our primary contribution is a highly accurate dropout prediction system that achieves an AUC-ROC of 0.908, accuracy of 87.4%, and precision of 0.981 using Random Forest classification. Using the “Adaptive Learning & Personalized Education Dataset,” we build predictive models to identify at-risk students, and these insights are then synthesized by an LLM to produce specific, coherent advice on intervention strategies. Since direct deployment with students was not feasible, we established a rigorous expert-based evaluation methodology to validate our system’s output. In this study, a panel of experienced educators assessed the LLM-generated recommendations across key dimensions of relevance, actionability, and pedagogical soundness. The results demonstrate that the experts consistently rated the AI-generated guidance as high-quality and contextually appropriate, confirming the viability of combining accurate dropout prediction with LLMs as a scalable tool for creating pedagogically valid support. This work not only presents a functional pipeline for generative educational advice but also provides a critical methodological blueprint for validating such AI-driven interventions in the absence of live student trials.