Scholarlink

<p>The adoption of technology in medical education, including the use of situational judgment tests (SJTs), reflects the field's evolving nature. The Association of American Medical Colleges (AAMC) Situational Judgment Test (SJT) is understood to be the gold standard to assess pre-professional competencies in prospective medical students. Thus, the impact of such an exam is extensive and students’ abilities to engage with it can deeply impact the future of health professions – in particular, medical education. This study aims to evaluate the performance of large language models (LLMs) in interpreting and responding to the AAMC SJT to understand the opportunities for its use and the obstacles that might exist. The study utilized the 2021 AAMC SJT practice exam, querying the LLMs to rate the effectiveness of various behavioral responses to each scenario. Both exact and similar agreement&#xa0;scores were calculated in accordance with AAMC’s guidelines. Statistical analysis involved descriptive statistics, logistic regression, and the calculation of Fleiss Kappa for inter-model agreement. LLMs at various stages of development were compared, with the idea that improvements might occur, a finding which would be of interest to educators using AI tools to evaluate SJTs, to prospective health professions trainees interested in preparing for such exams, and practicing professionals who aim to improve or fine tune their social intelligence-related skills. Our findings demonstrate that ChatGPT-4.0 (exact agreement&#xa0;score: 58.6%, similar&#xa0;agreement score: 71.2%) outperformed ChatGPT-3.5 (exact agreement&#xa0;score: 42.4%, similar agreement&#xa0;score: 61.6%) and Bard (exact agreement&#xa0;score: 37.6%, similar agreement&#xa0;score: 58.1%) with all LLMs tending towards making the same inaccuracies. Increased solution count in scenarios correlated with decreased accuracy across all models combined but these results fell short of statistical significance. Ultimately, our study contributes to understanding the capabilities and limitations of LLMs in the context of medical education assessments. ChatGPT-4.0's performance indicates a significant advancement in LLMs’ ability to interpret and respond to complex social situations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Large Language Models Take on the AAMC Situational Judgment Test: Evaluating Dilemma-Based Scenarios

  • 期刊论文

  • 发表日期：

    2025年4月4日

查看全文
  • Angelo Cadiente,
  • Jamie Chen,
  • Lora J. Kasselman,
  • Bryan Pilkington

摘要

The adoption of technology in medical education, including the use of situational judgment tests (SJTs), reflects the field's evolving nature. The Association of American Medical Colleges (AAMC) Situational Judgment Test (SJT) is understood to be the gold standard to assess pre-professional competencies in prospective medical students. Thus, the impact of such an exam is extensive and students’ abilities to engage with it can deeply impact the future of health professions – in particular, medical education. This study aims to evaluate the performance of large language models (LLMs) in interpreting and responding to the AAMC SJT to understand the opportunities for its use and the obstacles that might exist. The study utilized the 2021 AAMC SJT practice exam, querying the LLMs to rate the effectiveness of various behavioral responses to each scenario. Both exact and similar agreement scores were calculated in accordance with AAMC’s guidelines. Statistical analysis involved descriptive statistics, logistic regression, and the calculation of Fleiss Kappa for inter-model agreement. LLMs at various stages of development were compared, with the idea that improvements might occur, a finding which would be of interest to educators using AI tools to evaluate SJTs, to prospective health professions trainees interested in preparing for such exams, and practicing professionals who aim to improve or fine tune their social intelligence-related skills. Our findings demonstrate that ChatGPT-4.0 (exact agreement score: 58.6%, similar agreement score: 71.2%) outperformed ChatGPT-3.5 (exact agreement score: 42.4%, similar agreement score: 61.6%) and Bard (exact agreement score: 37.6%, similar agreement score: 58.1%) with all LLMs tending towards making the same inaccuracies. Increased solution count in scenarios correlated with decreased accuracy across all models combined but these results fell short of statistical significance. Ultimately, our study contributes to understanding the capabilities and limitations of LLMs in the context of medical education assessments. ChatGPT-4.0's performance indicates a significant advancement in LLMs’ ability to interpret and respond to complex social situations.