Chain of Thought Guided Fine-Tuning of Large Language Models for Rubric-Assisted Automated Essay Grading
摘要
Automated Essay Scoring (AES) is critical for providing timely feedback, reducing educators’ grading workloads, and improving student learning out-comes. Large language model (LLM), with their capacity for broad generalization, offer a new perspective in AES. However, current LLM-based AES systems often lack interpretability and consistent alignment with human scoring rubrics, which limits their practical application in educational envi-ronments. Addressing these limitations, we present COTScore+, a novel framework leveraging chain-of-thought (CoT) prompting and fine-tuning techniques to enhance the accuracy, interpretability, and rubric alignment of LLM-based AES approach. COTScore+ breaks down the assessment process into a sequence of reasoning steps that are explicitly aligned with the trait grading rubrics. This enables the system to formulate feedback that is readily interpretable, alongside providing trait-level scores that accurately reflect human evaluative reasoning. The experiments demonstrate that COTScore+surpasses robust baselines, including BERT and GPT-3.5, in generating feedback. This is evidenced by superior BLEU, ROUGE-L, and cosine similarity scores, as well as enhanced trait-level scoring, as indicated by quadratic weighted kappa (QWK). Moreover, the rubric-aligned CoT prompting facilitates the creation of feedback that more closely emulates human reasoning processes, a finding supported by both automatic metrics and human evaluations. These outcomes underscore the effectiveness of our method in delivering high-quality, human-aligned assessment feedback, thereby advancing the development of more transparent, dependable, and pedagogically sound AES systems suitable for practical implementation.