Background <p>The poor uptake of imaging referral guidelines in Europe results in a substantial amount of inappropriate computed tomography (CT) scans. Publicly available chatbots, ChatGPT and Gemini, offer an alternative for justifying real-world referrals. Recent research reports high ChatGPT accuracy when analysing American College of Radiology Appropriateness Criteria variants. We compared the chatbots’ performance in interpreting, justifying, and suggesting alternative imaging for unstructured adult brain CT referrals in accordance with the European Society of Radiology iGuide. Our prediction model for automated iGuide categorisation of referrals was also compared against the chatbots.</p> Methods <p>The iGuide justification of 143 real-world CT brain referrals, used to evaluate a prediction model, was analysed by two radiographers and radiologists. ChatGPT-4’s and Gemini’s imaging recommendations and pathology suspicions were compared with those of humans, with respect to referral completeness. Inter-rater reliability with κ statistics determined the agreement between entities.</p> Results <p>Chatbots’ performance was limited (κ = 0.3) but improved for more complete referrals. The prediction model outperformed the chatbots in justification analysis (κ = 0.853). The chatbots’ interpretations of complete referrals were highly consistent (49/52, 94.2%). The agreement regarding alternative imaging was high for both complete and ambiguous referrals, with ChatGPT and Gemini correctly identifying imaging modality and anatomical region in 83/96 (86.5%) and 81/96 (84.4%) cases, respectively.</p> Conclusion <p>The chatbots’ ability to analyse the justification of adult brain CT referrals is limited to complete referrals, unlike our prediction model. Further research is needed to confirm these findings for other types of CT scans and modalities.</p> Relevance statement <p>ChatGPT and Gemini exhibit potential in justifying free text brain CT referrals; however, further improvements are required to handle real-world referrals of varying quality.</p> Key Points <p><UnorderedList Mark="Bullet"> <ItemContent> <p>Custom prediction model’s justification analysis strongly aligns with iGuide and surpasses chatbots.</p> </ItemContent> <ItemContent> <p>Chatbots incorrectly justified almost one-half of all CT brain referrals.</p> </ItemContent> <ItemContent> <p>Chatbots have limited performance in justifying ambiguous CT brain referrals.</p> </ItemContent> <ItemContent> <p>Chatbot performance improved when referrals were detailed and included suspected pathology.</p> </ItemContent> </UnorderedList></p> Graphical Abstract <p></p>