Introduction This study aims to evaluate the accuracy and comprehensiveness of the information provided by ChatGPT-4, an artificial intelligence-based system, regarding ankylosing spondylitis (AS) from the perspectives of patients and physicians. Method In this cross-sectional study, 75 questions were asked of ChatGPT-4. These were the most frequently asked questions about AS on Google Trends (group 1), and questions derived from ASAS/EULAR recommendations (group 2 and group 3). Group 2 consisted of open-ended questions, and group 3 consisted of case questions. Two expert rheumatologists scored the responses for accuracy and comprehensiveness. A six-point Likert scale was used to assess accuracy, and a three-point scale for completeness. Results The accuracy and completeness scores analyzed in this study were found to be 5.32 ± 1.4 and 2.76 ± 0.5 for group 1, 5.36 ± 1.1 and 2.72 ± 0.45 for group 2, and 4.24 ± 1.96 and 2.36 ± 0.63 for group 3, respectively. There was a significant difference in accuracy and completeness scores between the groups (p = 0.044 and p = 0.019). Cohen’s kappa coefficient showed excellent agreement with values of 0.88 for accuracy and 0.90 for completeness. Conclusion While the responses to questions in groups 1 and 2 were satisfactory in terms of accuracy and comprehensiveness, the responses to complex case questions in group 3 were not sufficient. ChatGPT-4 appears to be a useful resource for patient education, but response mechanisms for complex clinical scenarios need to be improved. Furthermore, the potential for generating false or fabricated data should be considered; therefore, physicians should evaluate and verify responses. <Table Float="No" ID="Taba"> <tgroup cols="2"> <colspec align="left" colname="c1" colnum="1" /> <colspec align="left" colname="c2" colnum="2" /> <tbody> <row> <entry align="left" nameend="c2" namest="c1"> Key Points • ChatGPT-4 demonstrated high accuracy and comprehensiveness in addressing the information needs of patients and physicians regarding axial spondyloarthritis (AS), with satisfactory responses particularly for frequently asked questions and questions derived from ASAS/EULAR recommendations. • The quality of responses significantly decreased in the group of' case questions' involving more complex scenarios, indicating that the AI may be inadequate in clinical decision-making processes. • ChatGPT-4 can facilitate access to health information for patients, potentially reducing unnecessary clinic visits and serving as a primary source of information for those with limited access to healthcare professionals, thereby enhancing the overall effectiveness of patient care. • While ChatGPT-4 shows promise as a valuable tool for patient education, there is a clear need for the development of its response mechanisms, particularly for complex clinical scenarios. </entry> </row> </tbody> </tgroup> </Table>

Evaluation of artificial ıntelligence use in ankylosing spondylitis with ChatGPT-4: patient and physician perspectives

期刊论文
发表日期：

2025年9月11日

查看全文

Elif Altunel Kılınç,
Neşe Çabuk Çelik

摘要

Introduction

This study aims to evaluate the accuracy and comprehensiveness of the information provided by ChatGPT-4, an artificial intelligence-based system, regarding ankylosing spondylitis (AS) from the perspectives of patients and physicians.

Method

In this cross-sectional study, 75 questions were asked of ChatGPT-4. These were the most frequently asked questions about AS on Google Trends (group 1), and questions derived from ASAS/EULAR recommendations (group 2 and group 3). Group 2 consisted of open-ended questions, and group 3 consisted of case questions. Two expert rheumatologists scored the responses for accuracy and comprehensiveness. A six-point Likert scale was used to assess accuracy, and a three-point scale for completeness.

Results

The accuracy and completeness scores analyzed in this study were found to be 5.32 ± 1.4 and 2.76 ± 0.5 for group 1, 5.36 ± 1.1 and 2.72 ± 0.45 for group 2, and 4.24 ± 1.96 and 2.36 ± 0.63 for group 3, respectively. There was a significant difference in accuracy and completeness scores between the groups (p = 0.044 and p = 0.019). Cohen’s kappa coefficient showed excellent agreement with values of 0.88 for accuracy and 0.90 for completeness.

Conclusion

While the responses to questions in groups 1 and 2 were satisfactory in terms of accuracy and comprehensiveness, the responses to complex case questions in group 3 were not sufficient. ChatGPT-4 appears to be a useful resource for patient education, but response mechanisms for complex clinical scenarios need to be improved. Furthermore, the potential for generating false or fabricated data should be considered; therefore, physicians should evaluate and verify responses.

Key Points

• ChatGPT-4 demonstrated high accuracy and comprehensiveness in addressing the information needs of patients and physicians regarding axial spondyloarthritis (AS), with satisfactory responses particularly for frequently asked questions and questions derived from ASAS/EULAR recommendations.

• The quality of responses significantly decreased in the group of' case questions' involving more complex scenarios, indicating that the AI may be inadequate in clinical decision-making processes.

• ChatGPT-4 can facilitate access to health information for patients, potentially reducing unnecessary clinic visits and serving as a primary source of information for those with limited access to healthcare professionals, thereby enhancing the overall effectiveness of patient care.

• While ChatGPT-4 shows promise as a valuable tool for patient education, there is a clear need for the development of its response mechanisms, particularly for complex clinical scenarios.