Scholarlink

<p>This pilot study investigates whether artificial intelligence (AI) can effectively assess professional accounting examination scripts comparably to human expert evaluation. The research examines Claude 3.5, GPT-4o, Perplexity, and DeepSeek-V2 across nine subject areas of Ghanaian professional accounting examinations administered by the Institute of Chartered Accountants, Ghana (ICAG). A quantitative experimental design analysed 27 scripts across foundation, application, and professional levels, with each script assessed three times per condition per model, yielding 216 AI-based assessments for comparison with human examiner marks. Statistical analyses included mean absolute deviation (MAD), paired t-tests, chi-square and F-tests, and a novel Marking Scheme Responsiveness Index (MSRI) introduced to quantify AI responsiveness to structured criteria. Claude 3.5 demonstrated the closest alignment with the reference examiner’s marks under guided conditions, achieving a MAD of 4.1 points and a 46.1% improvement over its unguided condition (Cohen’s d = 1.12, 95% CI [0.52, 1.72]); however, as a single uniform prompt was applied across all models, this comparative advantage should be interpreted with caution given the well-documented sensitivity of large language model performance to prompt design. Conversely, GPT-4o showed deteriorating alignment under guided conditions, a counterintuitive finding that warrants further investigation. It is important to note that all AI scores are benchmarked against the marks of a single reference examiner, and the absence of human inter-examiner reliability data limits the interpretability of AI-human deviations. Results suggest assessment structure and model selection may be more influential than subject complexity in determining AI-human score alignment. Four contributions are made: the first empirical benchmark of four state-of-the-art large language models (LLMs) in professional accounting credentialing; operationalisation of the MSRI as a reusable metric; hypotheses challenging prevailing assumptions about AI in professional credentialing assessment; and a validated protocol for future large-scale investigation. All findings are hypothesis-generating, with confidence intervals reported throughout to guide replication planning.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A comparative pilot study of the accuracy consistency and marking scheme responsiveness of AI chatbots and human evaluators in professional accounting assessment

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月29日

查看全文
  • Samuel Koranteng Fianko,
  • Augustine Addo,
  • Kwasi Agyemang,
  • Frank Yao Gbadago,
  • Osei Adjaye-Gyamfi,
  • Olivia Quartey,
  • Frederick Agropah,
  • Rayyan Larttey,
  • Nathaniel Amoah,
  • Isaac Kwesi Nooni

摘要

This pilot study investigates whether artificial intelligence (AI) can effectively assess professional accounting examination scripts comparably to human expert evaluation. The research examines Claude 3.5, GPT-4o, Perplexity, and DeepSeek-V2 across nine subject areas of Ghanaian professional accounting examinations administered by the Institute of Chartered Accountants, Ghana (ICAG). A quantitative experimental design analysed 27 scripts across foundation, application, and professional levels, with each script assessed three times per condition per model, yielding 216 AI-based assessments for comparison with human examiner marks. Statistical analyses included mean absolute deviation (MAD), paired t-tests, chi-square and F-tests, and a novel Marking Scheme Responsiveness Index (MSRI) introduced to quantify AI responsiveness to structured criteria. Claude 3.5 demonstrated the closest alignment with the reference examiner’s marks under guided conditions, achieving a MAD of 4.1 points and a 46.1% improvement over its unguided condition (Cohen’s d = 1.12, 95% CI [0.52, 1.72]); however, as a single uniform prompt was applied across all models, this comparative advantage should be interpreted with caution given the well-documented sensitivity of large language model performance to prompt design. Conversely, GPT-4o showed deteriorating alignment under guided conditions, a counterintuitive finding that warrants further investigation. It is important to note that all AI scores are benchmarked against the marks of a single reference examiner, and the absence of human inter-examiner reliability data limits the interpretability of AI-human deviations. Results suggest assessment structure and model selection may be more influential than subject complexity in determining AI-human score alignment. Four contributions are made: the first empirical benchmark of four state-of-the-art large language models (LLMs) in professional accounting credentialing; operationalisation of the MSRI as a reusable metric; hypotheses challenging prevailing assumptions about AI in professional credentialing assessment; and a validated protocol for future large-scale investigation. All findings are hypothesis-generating, with confidence intervals reported throughout to guide replication planning.