Evaluating of Generative Artificial Intelligence Models for Automated Categorization of Programming Errors
摘要
Manual grading of programming assignments is time-consuming and often inconsistent, especially when students submit hundreds of code samples per term. This paper examines the performance of Generative Artificial Intelligence (GenAI) models in automatically categorizing programming errors in student Python code. The study focuses on six common error categories made by novice programmers: syntax, semantic, runtime, typing, control flow, and off-by-one. Seven GenAI models, including GPT-4o, Gemini 2.5, Claude 3.7, and DeepSeekV3, were used to classify these errors in student submissions. The dataset consists of 766 code submissions from Slovak-speaking students, allowing evaluation of model performance in a low-resource language context. Model outputs were compared against teacher annotations using F1 score, precision, recall, and statistical significance tests. Results show that GPT-4o consistently outperforms other models across most error categories, although all models perform poorly in low-frequency categories. These findings support the use of GenAI tools as possible preliminary evaluators in programming education. However, human oversight remains necessary for rare error cases or specific assignments. This paper offers insights into the practical application of GenAI in educational settings, where language and limited resources restrict access to high-end infrastructure.