Scholarlink

Background <p>The World Health Organization (WHO) 2023 Mutation Catalogue for <i>Mycobacterium tuberculosis</i> is a vital but complex knowledgebase for interpreting drug-resistant TB. Its size and density pose challenges for clinicians and researchers, hindering its accessibility. This study aimed to evaluate the potential of leading generative artificial intelligence (AI) models to facilitate natural language user interaction with the catalogue.</p> Results <p>Four prominent AI models (Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1) were benchmarked on general questions, mutation retrieval, and rule application tasks. Performance was measured by accuracy, completeness, clarity, source citation, and hallucinations. Google Gemini 2.5 Pro demonstrated the highest overall performance in accuracy and completeness, particularly on general queries and large dataset searches, with no hallucinations. DeepSeek R1 excelled in the logical task of applying grading rules to novel mutations. ChatGPT 4.1 provided clear responses but failed to cite sources properly. Perplexity AI showed the most variable performance and the highest rate of hallucinations.</p> Conclusions <p>This foundational proof-of-concept study demonstrates that while no current AI model is suitable for direct clinical use without further development, some models show significant potential to enhance the usability of complex biomedical documents. The performance differences highlight the importance of careful model selection and rigorous validation. With further refinement, models like Google Gemini 2.5 Pro could form the basis of a custom AI agent to support TB control efforts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking generative AI tools for interpretation of the WHO TB mutation catalogue

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月9日

查看全文
  • Miguel Moreno-Molina,
  • Anita Suresh,
  • Rebecca E. Colman,
  • Timothy C. Rodwell

摘要

Background

The World Health Organization (WHO) 2023 Mutation Catalogue for Mycobacterium tuberculosis is a vital but complex knowledgebase for interpreting drug-resistant TB. Its size and density pose challenges for clinicians and researchers, hindering its accessibility. This study aimed to evaluate the potential of leading generative artificial intelligence (AI) models to facilitate natural language user interaction with the catalogue.

Results

Four prominent AI models (Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1) were benchmarked on general questions, mutation retrieval, and rule application tasks. Performance was measured by accuracy, completeness, clarity, source citation, and hallucinations. Google Gemini 2.5 Pro demonstrated the highest overall performance in accuracy and completeness, particularly on general queries and large dataset searches, with no hallucinations. DeepSeek R1 excelled in the logical task of applying grading rules to novel mutations. ChatGPT 4.1 provided clear responses but failed to cite sources properly. Perplexity AI showed the most variable performance and the highest rate of hallucinations.

Conclusions

This foundational proof-of-concept study demonstrates that while no current AI model is suitable for direct clinical use without further development, some models show significant potential to enhance the usability of complex biomedical documents. The performance differences highlight the importance of careful model selection and rigorous validation. With further refinement, models like Google Gemini 2.5 Pro could form the basis of a custom AI agent to support TB control efforts.