Scholarlink

<p>The effective use of Large Language Models (LLMs) for generating coherent and informative content in specialized domains has largely been driven by the development of robust evaluation strategies. Based on this assumption, we introduce HemoQAL, a domain-specific question-and-answer (Q&amp;A) dataset on hemophilia, derived from recent scientific publications and clinical guidelines. Our main contribution lies in a fine-grained evaluation of the quality of LLM-generated content. First, we carried out a human evaluation in which medical experts assessed the factual accuracy and educational value of the generated Q&amp;A pairs. Second, we conducted a semantic similarity analysis to quantitatively evaluate the alignment between each Q&amp;A pair and its original source material. These lightweight, scalable semantic metrics offer an efficient alternative to more resource-intensive human or LLM-based evaluation pipelines. Our findings show that integrating expert review with semantic similarity measures improves the reliability and trustworthiness of LLM-generated medical content, contributing to the development of dependable AI tools in health informatics.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Fine-grained evaluation of a domain-specific Q&A dataset to support trustworthy medical language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月24日

查看全文
  • Rafael da C. Fonseca,
  • Ricardo A. Rios,
  • Rodrigo Castaldoni,
  • Adrielle A. Carvalho,
  • Tiago J. S. Lopes,
  • Caio L. B. Andrade,
  • Braian V. G. Bispo,
  • Laís R. Mota,
  • Tatiane N. Rios

摘要

The effective use of Large Language Models (LLMs) for generating coherent and informative content in specialized domains has largely been driven by the development of robust evaluation strategies. Based on this assumption, we introduce HemoQAL, a domain-specific question-and-answer (Q&A) dataset on hemophilia, derived from recent scientific publications and clinical guidelines. Our main contribution lies in a fine-grained evaluation of the quality of LLM-generated content. First, we carried out a human evaluation in which medical experts assessed the factual accuracy and educational value of the generated Q&A pairs. Second, we conducted a semantic similarity analysis to quantitatively evaluate the alignment between each Q&A pair and its original source material. These lightweight, scalable semantic metrics offer an efficient alternative to more resource-intensive human or LLM-based evaluation pipelines. Our findings show that integrating expert review with semantic similarity measures improves the reliability and trustworthiness of LLM-generated medical content, contributing to the development of dependable AI tools in health informatics.