Scholarlink

<p>This article aims to evaluate the quality of responses provided by AI systems ChatGPT-4o and Gemini 1.5 to frequently asked patient questions regarding pectus deformity. Method: In this cross-sectional survey study, 36 frequently asked questions about pectus surgery were posed using a new Google account with no search history. The responses to the questions were recorded using the ChatGPT-4o and Gemini 1.5 AI programs. The responses were rated by 10 surgeons specialized in pectus surgery using a rating scale for relevance, accuracy, clarity, and completeness. The intraclass correlation coefficient (ICC) was used for interrater reliability (IRR) analysis of the evaluators’ responses. Results: In the study, the average score for relevance was 4.79 for ChatGPT-4o and 4.86 for Gemini 1.5. According to the accuracy criterion, ChatGPT-4o had an average of 4.59, while Gemini 1.5 had 4.64. Regarding clarity, ChatGPT-4o scored an average of 4.61, and Gemini1.5 scored 4.75. No statistically significant difference was found between the responses of the two models in terms of relevance, accuracy, clarity, and completeness. Additionally, no statistical difference was found in the IRR analyses during the evaluation process. The reliability of the ChatGPT-4o and Gemini 1.5 language models was compared based on various criteria. Conclusion: As a result of this study, we believe that AI applications have a high potential in informing patients about pectus surgery. However, they cannot replace professional medical advice, and therefore, we believe that patients should consult experts to verify AI responses.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating the accuracy and patient perception of AI-generated answers on pectus surgery

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年11月25日

查看全文
  • Esra Yamansavcı Sirzai,
  • Celal Bugra Sezen,
  • Gizem Ozcibik Isik,
  • Mustafa Yuksel

摘要

This article aims to evaluate the quality of responses provided by AI systems ChatGPT-4o and Gemini 1.5 to frequently asked patient questions regarding pectus deformity. Method: In this cross-sectional survey study, 36 frequently asked questions about pectus surgery were posed using a new Google account with no search history. The responses to the questions were recorded using the ChatGPT-4o and Gemini 1.5 AI programs. The responses were rated by 10 surgeons specialized in pectus surgery using a rating scale for relevance, accuracy, clarity, and completeness. The intraclass correlation coefficient (ICC) was used for interrater reliability (IRR) analysis of the evaluators’ responses. Results: In the study, the average score for relevance was 4.79 for ChatGPT-4o and 4.86 for Gemini 1.5. According to the accuracy criterion, ChatGPT-4o had an average of 4.59, while Gemini 1.5 had 4.64. Regarding clarity, ChatGPT-4o scored an average of 4.61, and Gemini1.5 scored 4.75. No statistically significant difference was found between the responses of the two models in terms of relevance, accuracy, clarity, and completeness. Additionally, no statistical difference was found in the IRR analyses during the evaluation process. The reliability of the ChatGPT-4o and Gemini 1.5 language models was compared based on various criteria. Conclusion: As a result of this study, we believe that AI applications have a high potential in informing patients about pectus surgery. However, they cannot replace professional medical advice, and therefore, we believe that patients should consult experts to verify AI responses.