Scholarlink

Objective <p>This study aimed to compare the reliability of references generated by three different AI-based conversational agents (ChatGPT, Gemini, Perplexity) within the rotator cuff literature.</p> Methods <p>Designed as a cross-sectional study, 30 subtopics were posed to each bot in two formats (letter to the editor and original article), resulting in a total of 3150 references. Reference reliability was assessed using the Reference Hallucination Score (RHS), which included the following criteria: existence/verifiability, bibliographic accuracy, PMID validity, and topical relevance.</p> Results <p>When all formats were analyzed together, ChatGPT had the lowest mean RHS (1.81 ± 3.40) and thus emerged as the most reliable model. Gemini scored 4.01 ± 4.89, while Perplexity had the highest score at 6.51 ± 4.89 (<i>p</i> &lt; 0.001). In the letter format, ChatGPT, Gemini, and Perplexity scored 1.81 ± 3.20, 3.81 ± 4.83, and 6.43 ± 4.78, respectively. In the article format, ChatGPT scored 4.02 ± 1.70, Gemini 4.13 ± 1.57, and Perplexity 6.31 ± 1.68. ChatGPT was more reliable than Gemini and significantly superior to Perplexity.</p> Conclusion <p>AI-based conversational agents may play a supportive role in academic writing; however, they exhibit critical limitations in reference accuracy. While ChatGPT was relatively the most reliable model, Perplexity had the highest hallucination rate. Researchers must refrain from using references generated by these tools without verification, as doing so poses significant concerns for scientific integrity and ethics.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Reference Hallucination in AI-Assisted Academic Writing: A Comparative Analysis of ChatGPT, Gemini, and Perplexity in Rotator Cuff Literature

  • 期刊论文

  • 发表日期：

    2026年5月11日

查看全文
  • İlhan Celil Özbek,
  • Fatih Bağcıer

摘要

Objective

This study aimed to compare the reliability of references generated by three different AI-based conversational agents (ChatGPT, Gemini, Perplexity) within the rotator cuff literature.

Methods

Designed as a cross-sectional study, 30 subtopics were posed to each bot in two formats (letter to the editor and original article), resulting in a total of 3150 references. Reference reliability was assessed using the Reference Hallucination Score (RHS), which included the following criteria: existence/verifiability, bibliographic accuracy, PMID validity, and topical relevance.

Results

When all formats were analyzed together, ChatGPT had the lowest mean RHS (1.81 ± 3.40) and thus emerged as the most reliable model. Gemini scored 4.01 ± 4.89, while Perplexity had the highest score at 6.51 ± 4.89 (p < 0.001). In the letter format, ChatGPT, Gemini, and Perplexity scored 1.81 ± 3.20, 3.81 ± 4.83, and 6.43 ± 4.78, respectively. In the article format, ChatGPT scored 4.02 ± 1.70, Gemini 4.13 ± 1.57, and Perplexity 6.31 ± 1.68. ChatGPT was more reliable than Gemini and significantly superior to Perplexity.

Conclusion

AI-based conversational agents may play a supportive role in academic writing; however, they exhibit critical limitations in reference accuracy. While ChatGPT was relatively the most reliable model, Perplexity had the highest hallucination rate. Researchers must refrain from using references generated by these tools without verification, as doing so poses significant concerns for scientific integrity and ethics.