Scholarlink

<p>Systematic reviews in medical education often classify outcomes using the Kirkpatrick framework, but manual coding is time-consuming and subjective. We conducted a proof-of-concept study testing ChatGPT (GPT-5, August 2025 release) on 32 full-text articles from a published systematic review of sepsis education. Agreement with human-coded outcomes was modest: 50% percent agreement, unweighted κ = 0.170 (95% CI 0.000–0.458), weighted κ = 0.351 (95% CI 0.074–0.629). Most disagreements were between adjacent levels.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Can Large Language Models Replicate Systematic Review Outcome Classifications in Medical Education? A Pilot Study Using Kirkpatrick Levels

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月16日

查看全文
  • Giuliano Romano,
  • Emilio Romano,
  • Michelle Rau

摘要

Systematic reviews in medical education often classify outcomes using the Kirkpatrick framework, but manual coding is time-consuming and subjective. We conducted a proof-of-concept study testing ChatGPT (GPT-5, August 2025 release) on 32 full-text articles from a published systematic review of sepsis education. Agreement with human-coded outcomes was modest: 50% percent agreement, unweighted κ = 0.170 (95% CI 0.000–0.458), weighted κ = 0.351 (95% CI 0.074–0.629). Most disagreements were between adjacent levels.