Scholarlink

<p>Generative Pre-trained Transformers (GPT) has ushered in a transformative shift in the landscape of automated text generation. These models use artificial neural networks to produce responses as per human prompts. These GPTs are now widely used in many NLP tools such as ChatGPT, Gemini, Pi, and Perplexity. As these systems become more common in academic and knowledge-based work, it is important to carefully examine the authenticity and reliability of their outputs. This study investigates the text-generation capabilities of these GPT-based tools by employing the cosine similarity index to compare their outputs against ground truth definitions sourced from the Britannica Encyclopedia. Definitions of selected concepts in science and social science were analyzed to ascertain the alignment of GPT-generated texts with established knowledge. The results show clear differences in how closely the four GPT-based systems matched Britannica definitions. Perplexity achieved the highest similarity with the least variability, followed closely by Pi and ChatGPT, while Gemini consistently produced the lowest similarity scores. The analysis of variance confirmed the statistical significance of these differences. The post hoc testing showed that Gemini formed a separate group due to its lower alignment with ground-truth definitions. These findings highlight that although GPT tools provide reasonably accurate information, their outputs still vary from ground truths. Overall, Perplexity Pi, and ChatGPT were closer to ground truth than Gemini, indicating the need for further improvement in Gemini’s performance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Measuring accuracy of AI generated definitions using cosine similarity index across select GPT models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月5日

查看全文
  • Niyasha Patra,
  • Suraj Sharma,
  • Nilanjan Ray,
  • Debkumar Bera

摘要

Generative Pre-trained Transformers (GPT) has ushered in a transformative shift in the landscape of automated text generation. These models use artificial neural networks to produce responses as per human prompts. These GPTs are now widely used in many NLP tools such as ChatGPT, Gemini, Pi, and Perplexity. As these systems become more common in academic and knowledge-based work, it is important to carefully examine the authenticity and reliability of their outputs. This study investigates the text-generation capabilities of these GPT-based tools by employing the cosine similarity index to compare their outputs against ground truth definitions sourced from the Britannica Encyclopedia. Definitions of selected concepts in science and social science were analyzed to ascertain the alignment of GPT-generated texts with established knowledge. The results show clear differences in how closely the four GPT-based systems matched Britannica definitions. Perplexity achieved the highest similarity with the least variability, followed closely by Pi and ChatGPT, while Gemini consistently produced the lowest similarity scores. The analysis of variance confirmed the statistical significance of these differences. The post hoc testing showed that Gemini formed a separate group due to its lower alignment with ground-truth definitions. These findings highlight that although GPT tools provide reasonably accurate information, their outputs still vary from ground truths. Overall, Perplexity Pi, and ChatGPT were closer to ground truth than Gemini, indicating the need for further improvement in Gemini’s performance.