Assessing the accuracy of ChatGPT responses to caregiver questions in pediatric neurology: an observational study
摘要
ChatGPT has seen increased use by caregivers as a source of medical information. Pediatric neurology has added complexities because there are various disorders, changing diagnoses and potential consequences of misinformation. To evaluate the accuracy, comprehensiveness and reproducibility of ChatGPT responses to frequently asked pediatric neurology questions. In total, 105 unique questions across seven pediatric neurology subspecialties were included. Every question was input twice into ChatGPT (GPT-5) during separate sessions. Two certified pediatric neurologists reviewed the responses and rated them on a 4-point scale: 1-complete and accurate,2- correct but not complete,3- partially incorrect,4-not correct. Reproducibility was considered to be agreement between grading categories (grade 1–2 and grade 3–4) throughout repeated responses. 60.95%(64/105) of responses were rated as comprehensive, while 31.43%(33/105) were correct but incomplete; fully incorrect responses were uncommon 3.8%(4/105). Accuracy and reproducibility were highest for structured conditions such as cerebral palsy and movement disorders, and lower for more heterogeneous domains including neurodevelopmental delay and epilepsy. Overall reproducibility was high 95.2%(100/105). ChatGPT mostly delivers accurate information in pediatric neurology, with variable depth of response depending on the disorder. It is most appropriate for caregiver education with oversight by the clinician.