Scholarlink

<p>To evaluate how different prompting strategies influence the diagnostic accuracy, stability, and interpretability of a multimodal large language model (LLM) (ChatGPT-5.1)&#xa0;in oral lesion assessment and to introduce the Prompt Performance Index (PPI), a novel robustness metric that quantifies the performance stability across clinically relevant subgroups. A total of 300 biopsy-confirmed clinical vignettes were assessed using three distinct prompting strategies: a Context-Focused Prompt (CFP), an Evidence-Guided Prompt (EGP), and a Consistency-Optimized Prompt (COP). Diagnostic accuracy (Top-1 and Top-3), confidence calibration, explanation quality, and computational efficiency were compared. The PPI, defined as mean diagnostic accuracy penalized by variability across subgroups, was developed to characterize robustness across diagnostic difficulty tiers and lesion categories. Multivariable regression models examined the independent effects of prompt structure, case difficulty, and lesion type. COP achieved the highest numerical Top-1 accuracy (72.0%) and the greatest performance stability across difficulty levels and lesion categories. Confidence calibration and explanation quality were also superior for COP (<i>p</i> &lt; 0.0001). Multivariable regression analysis confirmed that both prompt design and diagnostic difficulty significantly influenced model accuracy. The PPI differentiated prompting strategies by revealing variation in diagnostic performance stability not captured by accuracy metrics alone, with COP yielding the highest PPI values under both Top-1 and Top-3 performance. Prompt engineering meaningfully affects multimodal LLM diagnostic behavior in oral medicine. The proposed PPI provides a novel and clinically interpretable framework for evaluating diagnostic robustness across heterogeneous clinical conditions, complementing conventional accuracy-based assessments and enabling more dependable diagnostic workflows with AI assistance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Prompt engineering shapes diagnostic accuracy and explanation quality of LLM in oral lesion diagnosis: a prospective, expert-blinded benchmark study

  • 期刊论文

  • 发表日期：

    2026年4月12日

查看全文
  • Fatma E. A. Hassanein,
  • Suzan S. Ibrahim,
  • Saygo Tomo,
  • Abdulaziz Alsahhaf,
  • Asmaa Abou-Bakr

摘要

To evaluate how different prompting strategies influence the diagnostic accuracy, stability, and interpretability of a multimodal large language model (LLM) (ChatGPT-5.1) in oral lesion assessment and to introduce the Prompt Performance Index (PPI), a novel robustness metric that quantifies the performance stability across clinically relevant subgroups. A total of 300 biopsy-confirmed clinical vignettes were assessed using three distinct prompting strategies: a Context-Focused Prompt (CFP), an Evidence-Guided Prompt (EGP), and a Consistency-Optimized Prompt (COP). Diagnostic accuracy (Top-1 and Top-3), confidence calibration, explanation quality, and computational efficiency were compared. The PPI, defined as mean diagnostic accuracy penalized by variability across subgroups, was developed to characterize robustness across diagnostic difficulty tiers and lesion categories. Multivariable regression models examined the independent effects of prompt structure, case difficulty, and lesion type. COP achieved the highest numerical Top-1 accuracy (72.0%) and the greatest performance stability across difficulty levels and lesion categories. Confidence calibration and explanation quality were also superior for COP (p < 0.0001). Multivariable regression analysis confirmed that both prompt design and diagnostic difficulty significantly influenced model accuracy. The PPI differentiated prompting strategies by revealing variation in diagnostic performance stability not captured by accuracy metrics alone, with COP yielding the highest PPI values under both Top-1 and Top-3 performance. Prompt engineering meaningfully affects multimodal LLM diagnostic behavior in oral medicine. The proposed PPI provides a novel and clinically interpretable framework for evaluating diagnostic robustness across heterogeneous clinical conditions, complementing conventional accuracy-based assessments and enabling more dependable diagnostic workflows with AI assistance.