<p>To systematically evaluate and compare the performance of large language models (LLMs) in answering osteoporosis-related frequently asked questions (FAQs) derived from international osteoporosis-related associations. A standardized question bank was constructed based on FAQs summarized from three international osteoporosis-related associations. Six LLMs were prompted to generate responses to all questions under uniform conditions. Two osteoporosis experts independently and blindly evaluated all responses using 5-point Likert scales for accuracy and comprehensiveness. Inter-rater reliability was assessed using Cohen’s κ coefficient. Nonparametric statistical analyses, including Kruskal-Wallis tests with Dunn’s post hoc comparisons and Wilcoxon signed-rank tests, were performed to compare models’ performance. A total of 528 responses were generated from 88 association-based questions. Inter-rater agreement was good for both accuracy and comprehensiveness (κ = 0.668 and 0.702, respectively). Significant differences were observed among models, with Claude Sonnet 4.5 demonstrating the highest overall performance on association-based FAQs. Although most models showed solid medical knowledge and basic reasoning ability, substantial variability was observed across question types and prompt formulations. Current LLMs demonstrate promising potential as auxiliary tools for osteoporosis-related health management. However, it is necessary to optimize the strategies for integrating medical knowledge and designing prompts, so as to better improve the accessibility of its clinical application.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance of Large Language Models in Answering Osteoporosis-Related Frequently Asked Questions: A Systematic Comparative Evaluation Based on International Associations

  • Xuhui Wu,
  • Jing Li,
  • Shuying Liu,
  • Yue Li,
  • Qinyi Wang,
  • Chuo Luo,
  • Linke Shi,
  • Yanfen Zheng,
  • Lingqiong Meng,
  • Min Fan,
  • Zhifeng Sheng,
  • Na Ouyang

摘要

To systematically evaluate and compare the performance of large language models (LLMs) in answering osteoporosis-related frequently asked questions (FAQs) derived from international osteoporosis-related associations. A standardized question bank was constructed based on FAQs summarized from three international osteoporosis-related associations. Six LLMs were prompted to generate responses to all questions under uniform conditions. Two osteoporosis experts independently and blindly evaluated all responses using 5-point Likert scales for accuracy and comprehensiveness. Inter-rater reliability was assessed using Cohen’s κ coefficient. Nonparametric statistical analyses, including Kruskal-Wallis tests with Dunn’s post hoc comparisons and Wilcoxon signed-rank tests, were performed to compare models’ performance. A total of 528 responses were generated from 88 association-based questions. Inter-rater agreement was good for both accuracy and comprehensiveness (κ = 0.668 and 0.702, respectively). Significant differences were observed among models, with Claude Sonnet 4.5 demonstrating the highest overall performance on association-based FAQs. Although most models showed solid medical knowledge and basic reasoning ability, substantial variability was observed across question types and prompt formulations. Current LLMs demonstrate promising potential as auxiliary tools for osteoporosis-related health management. However, it is necessary to optimize the strategies for integrating medical knowledge and designing prompts, so as to better improve the accessibility of its clinical application.