Scholarlink

<p>Large language models (LLMs) have demonstrated substantial potential in medical practice; however, no systematic framework currently exists for evaluating their ethical responses. This study aimed to develop a novel evaluation corpus reflecting real-world clinical ethical challenges and assess the ethical responses of LLMs using psychometric methods. We constructed an interactive corpus comprising 50 medical ethics scenarios, with each option scored based on the degree of violation of established moral rules. Item response theory (IRT) was applied to comprehensively examine the reliability, factor structure, measurement precision, and item characteristics of the corpus. Furthermore, a virtual patient dataset was created by manipulating social attributes such as age, gender, race, and medical history to explore biases in the ethical responses of LLMs. IRT analyses demonstrated that the abstract concept of <i>ethicality</i> can be quantified, allowing direct comparison between human physicians and LLMs. GPT-4 exhibited the highest ethicality scores, whereas GPT-3.5 and Llama 2 consistently scored lower. Analyses using virtual patients further revealed that the ethical decisions of the LLMs systematically varied with patient attributes, with both the Japanese and English versions showing reduced ethicality toward older patients. This study represents the first attempt to quantify ethicality using a purpose-built corpus and IRT, enabling a direct comparison of LLMs and humans. The proposed approach provides an objective framework for validating the clinical applicability of LLMs and detecting potential biases in their ethical responses.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Quantifying ethical response in LLMs for medicine: corpus development, item response theory-based validation, and bias analysis toward patient attributes

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月28日

查看全文
  • Memi Ebara,
  • Yoshimasa Kawazoe,
  • Tomohisa Seki,
  • Emiko Shinohara,
  • Eisuke Nakazawa,
  • Kazuhiko Ohe

摘要

Large language models (LLMs) have demonstrated substantial potential in medical practice; however, no systematic framework currently exists for evaluating their ethical responses. This study aimed to develop a novel evaluation corpus reflecting real-world clinical ethical challenges and assess the ethical responses of LLMs using psychometric methods. We constructed an interactive corpus comprising 50 medical ethics scenarios, with each option scored based on the degree of violation of established moral rules. Item response theory (IRT) was applied to comprehensively examine the reliability, factor structure, measurement precision, and item characteristics of the corpus. Furthermore, a virtual patient dataset was created by manipulating social attributes such as age, gender, race, and medical history to explore biases in the ethical responses of LLMs. IRT analyses demonstrated that the abstract concept of ethicality can be quantified, allowing direct comparison between human physicians and LLMs. GPT-4 exhibited the highest ethicality scores, whereas GPT-3.5 and Llama 2 consistently scored lower. Analyses using virtual patients further revealed that the ethical decisions of the LLMs systematically varied with patient attributes, with both the Japanese and English versions showing reduced ethicality toward older patients. This study represents the first attempt to quantify ethicality using a purpose-built corpus and IRT, enabling a direct comparison of LLMs and humans. The proposed approach provides an objective framework for validating the clinical applicability of LLMs and detecting potential biases in their ethical responses.