Scholarlink

<p>The assessment of Task Response (TR) in high-stakes exams like IELTS is plagued by scorer subjectivity, conceptual vagueness, and limited pedagogical utility, which Automated Writing Evaluation (AWE) tools, including “black-box” Generative AI, have not consistently addressed in formative contexts. This challenge is further compounded by variable levels of feedback literacy among learners. To bridge these gaps, this exploratory, proof-of-concept study develops an interpretable, five-dimensional TR framework (Coverage, Opinion, Claim, Grounds, Penalty) through thematic analysis of 28 official IELTS essays (Bands 5.0–7.0). The proposed framework demonstrates promising inter-rater reliability (Cohen’s kappa = 0.827) and strong internal content validity (S-CVI = 0.92) within the context of official descriptors. Preliminary results from Kruskal-Wallis (<i>p</i> &lt; 0.001) and Spearman correlations (<i>r</i> &gt; 0.90 for Claim and Grounds) suggest its potential to discriminate between proficiency levels. By mapping this framework onto GPT-4 via optimized prompts, the study shifts AI’s role from an opaque scorer (Mean Absolute Error, MAE = 0.54; correlation with human scores, <i>r</i> = 0.39) toward a structured diagnostic feedback tool (MAE = 0.41; <i>r</i> = 0.61). While these results represent a promising proof-of-concept for formative use, the levels of agreement do not yet meet the requirements for high-stakes summative assessment. Given the limited sample size and exploratory design, these findings should be interpreted cautiously as preliminary evidence for framework viability. The study highlights the framework’s potential to enhance pedagogical transparency and support self-regulation through ZPD-aligned interventions. Future research is needed to validate the framework in larger and more diverse samples and to examine its practical impact in classroom settings.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An interpretable framework for task response in IELTS writing: integrating thematic analysis, argumentation dimensions, and GPT-4 for educational feedback

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月12日

查看全文
  • Weiqiang Li,
  • Nur Ainil Sulaiman,
  • Mohd Effendi Ewan Mohd Matore,
  • Xin Ke

摘要

The assessment of Task Response (TR) in high-stakes exams like IELTS is plagued by scorer subjectivity, conceptual vagueness, and limited pedagogical utility, which Automated Writing Evaluation (AWE) tools, including “black-box” Generative AI, have not consistently addressed in formative contexts. This challenge is further compounded by variable levels of feedback literacy among learners. To bridge these gaps, this exploratory, proof-of-concept study develops an interpretable, five-dimensional TR framework (Coverage, Opinion, Claim, Grounds, Penalty) through thematic analysis of 28 official IELTS essays (Bands 5.0–7.0). The proposed framework demonstrates promising inter-rater reliability (Cohen’s kappa = 0.827) and strong internal content validity (S-CVI = 0.92) within the context of official descriptors. Preliminary results from Kruskal-Wallis (p < 0.001) and Spearman correlations (r > 0.90 for Claim and Grounds) suggest its potential to discriminate between proficiency levels. By mapping this framework onto GPT-4 via optimized prompts, the study shifts AI’s role from an opaque scorer (Mean Absolute Error, MAE = 0.54; correlation with human scores, r = 0.39) toward a structured diagnostic feedback tool (MAE = 0.41; r = 0.61). While these results represent a promising proof-of-concept for formative use, the levels of agreement do not yet meet the requirements for high-stakes summative assessment. Given the limited sample size and exploratory design, these findings should be interpreted cautiously as preliminary evidence for framework viability. The study highlights the framework’s potential to enhance pedagogical transparency and support self-regulation through ZPD-aligned interventions. Future research is needed to validate the framework in larger and more diverse samples and to examine its practical impact in classroom settings.