Scholarlink

Background <p>Chronic obstructive pulmonary disease (COPD) remains an underestimated and underdiagnosed condition due to low disease awareness. Generative Artificial Intelligence (AI) chatbots are convenient and accessible sources of medical information, but evaluation of the quality of answers provided by patient-generated questions about COPD has not been performed to date.</p> Objective <p>To assess and compare accuracy, comprehensiveness, understandability and reliability of different AI chatbots in response to patient-generated questions on the clinical management of COPD.</p> Methods <p>A cross-sectional study was conducted in collaboration with the European Respiratory Society (ERS), the European Lung Foundation (ELF), and the ERS CONNECT Clinical Research Collaboration (CRC). Fifteen real questions formulated by ELF COPD patient representatives were divided into three difficulty tiers (easy, medium, difficult) and submitted to ChatGPT (version 3.5), Bard, and Copilot. Experts assessed accuracy and comprehensiveness on a 0–10 scale; patients assessed understandability using the same scale. Reliability was assessed by two investigators. Reviewers were blinded to which AI system generated the answers, and only those who completed all evaluations were included in the analysis.</p> Results <p>ChatGPT responses were the most reliable (14/15), followed by Copilot (12/15) and Bard (11/15). ChatGPT scored higher for accuracy (8.0 [7.0 – 9.0]) and comprehensiveness (8.0 [6.8 – 9.0]) than Bard (6.0 [5.0 – 8.0] and 6.0 [5.0 – 7.0]) and Copilot (6.0 [5.0 – 7.3] and 6.0 [5.0 – 8.0]) (both <i>P</i> &lt; 0.001). Understandability was similar across all software (ChatGPT: 8.0 [8.0–10.0]; Bard: 9.0 [8.0–10.0]; Copilot: 9.0 [8.0–10.0]) (<i>P</i> = 0.53). No significant effect was detected according to the difficulty of the question.</p> Conclusion <p>Our findings suggest that AI chatbots, particularly ChatGPT, can provide accurate, comprehensive and understandable answers to patients’ questions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Accuracy, comprehensiveness and understandability of AI-generated answers to questions from people with COPD: the AIR-COPD Study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年12月16日

查看全文
  • Mattia Nigro,
  • Greta E. Behring,
  • Andrea Aliverti,
  • Alessandra Angelucci,
  • Anita Kay Simonds,
  • Antonio Anzueto,
  • Peter Martin Calverley,
  • Francesco Amati,
  • Anna Stainer,
  • Apostolos Bossios,
  • Hilary Pinnock,
  • Jeanette Boyd,
  • Pippa Powell,
  • Stefano Aliberti,
  • Antonio Spanevello,
  • Marco Vanetti,
  • David M. G. Halpin,
  • Khanh Le Quoc Tran,
  • Maria Elia Gomez-Merino,
  • Deepak Muthreja,
  • Boudewijn J. H. Dierick,
  • Elene Khurtsidze,
  • Vishakha Kalpesh Kapadia,
  • Amalia Panagiotou,
  • Miguel Gallego,
  • Stavros Tryfon,
  • Efthymia Papadopoulou,
  • Carlos Figueiredo,
  • Shailesh Balasaheb Kolekar,
  • Pradeesh Sivapalan,
  • Pedro J. Marcos

摘要

Background

Chronic obstructive pulmonary disease (COPD) remains an underestimated and underdiagnosed condition due to low disease awareness. Generative Artificial Intelligence (AI) chatbots are convenient and accessible sources of medical information, but evaluation of the quality of answers provided by patient-generated questions about COPD has not been performed to date.

Objective

To assess and compare accuracy, comprehensiveness, understandability and reliability of different AI chatbots in response to patient-generated questions on the clinical management of COPD.

Methods

A cross-sectional study was conducted in collaboration with the European Respiratory Society (ERS), the European Lung Foundation (ELF), and the ERS CONNECT Clinical Research Collaboration (CRC). Fifteen real questions formulated by ELF COPD patient representatives were divided into three difficulty tiers (easy, medium, difficult) and submitted to ChatGPT (version 3.5), Bard, and Copilot. Experts assessed accuracy and comprehensiveness on a 0–10 scale; patients assessed understandability using the same scale. Reliability was assessed by two investigators. Reviewers were blinded to which AI system generated the answers, and only those who completed all evaluations were included in the analysis.

Results

ChatGPT responses were the most reliable (14/15), followed by Copilot (12/15) and Bard (11/15). ChatGPT scored higher for accuracy (8.0 [7.0 – 9.0]) and comprehensiveness (8.0 [6.8 – 9.0]) than Bard (6.0 [5.0 – 8.0] and 6.0 [5.0 – 7.0]) and Copilot (6.0 [5.0 – 7.3] and 6.0 [5.0 – 8.0]) (both P < 0.001). Understandability was similar across all software (ChatGPT: 8.0 [8.0–10.0]; Bard: 9.0 [8.0–10.0]; Copilot: 9.0 [8.0–10.0]) (P = 0.53). No significant effect was detected according to the difficulty of the question.

Conclusion

Our findings suggest that AI chatbots, particularly ChatGPT, can provide accurate, comprehensive and understandable answers to patients’ questions.