Scholarlink

Purpose <p>To propose a novel, standardized, and safety-centered tool for Large Language Models (LLMs) evaluation.</p> Methods <p>The Medical Evaluation of Large Language Model Answers Questionnaire (MELMA-Q) was developed as a 30-item clinician-rated instrument spanning seven domains: Medical Accuracy/Groundedness, Clinical Reasoning/Management, Safety/Ethics/Trustworthiness, Linguistic Quality/Semantic Fidelity, Understandability/Literacy Adaptation, Usefulness/Decision Support, and Performance/Answer Behavior. The MELMA Clinical Acceptability Framework (MELMA-CAF) is a two-tier system that incorporates a non-compensatory safety gate and weighted scoring. Five standardized otolaryngology scenarios were posed to three LLMs (ChatGPT 5.2, Gemini Flash 3, DeepSeek v3.2), generating 15 responses, which were independently scored by five blinded ENT specialists. A web-based implementation (MELMA-W) operationalized rubric-based scoring and was compared with clinician ratings.</p> Results <p>All responses passed Tier A safety screening. Mean total MELMA-Q scores ranged from 72.4 to 85.6 across models; inter-rater reliability was excellent (ICC 0.89; 95% CI 0.84–0.93). MELMA-W validation using paired model × domain observations showed systematically higher clinician scores (bias of 0.804 Likert points).</p> Conclusions <p>MELMA-Q and MELMA-W provide a structured, safety-centered pilot framework for evaluating LLM-generated medical responses in otolaryngology; however, broader validation across larger datasets, additional raters, and other clinical specialties remains required.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

“MELMA” in otolaryngology: Medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment

  • 期刊论文

  • 发表日期：

    2026年6月22日

查看全文
  • Natig Ahmadov,
  • Antiga Muradova,
  • Aynur Aliyeva

摘要

Purpose

To propose a novel, standardized, and safety-centered tool for Large Language Models (LLMs) evaluation.

Methods

The Medical Evaluation of Large Language Model Answers Questionnaire (MELMA-Q) was developed as a 30-item clinician-rated instrument spanning seven domains: Medical Accuracy/Groundedness, Clinical Reasoning/Management, Safety/Ethics/Trustworthiness, Linguistic Quality/Semantic Fidelity, Understandability/Literacy Adaptation, Usefulness/Decision Support, and Performance/Answer Behavior. The MELMA Clinical Acceptability Framework (MELMA-CAF) is a two-tier system that incorporates a non-compensatory safety gate and weighted scoring. Five standardized otolaryngology scenarios were posed to three LLMs (ChatGPT 5.2, Gemini Flash 3, DeepSeek v3.2), generating 15 responses, which were independently scored by five blinded ENT specialists. A web-based implementation (MELMA-W) operationalized rubric-based scoring and was compared with clinician ratings.

Results

All responses passed Tier A safety screening. Mean total MELMA-Q scores ranged from 72.4 to 85.6 across models; inter-rater reliability was excellent (ICC 0.89; 95% CI 0.84–0.93). MELMA-W validation using paired model × domain observations showed systematically higher clinician scores (bias of 0.804 Likert points).

Conclusions

MELMA-Q and MELMA-W provide a structured, safety-centered pilot framework for evaluating LLM-generated medical responses in otolaryngology; however, broader validation across larger datasets, additional raters, and other clinical specialties remains required.