While Large Language Models (LLMs) demonstrate excellent performance, their output depends on pre-trained knowledge. It has been pointed out that LLMs have various problems, such as generating outdated or incorrect information and they cannot always generate accurate answers. Retrieval-Augmented Generation (RAG) is a technical means that can mitigate the aforementioned problems by connecting an external database to LLM and retrieving relevant information from it, thereby making it possible to utilize a piece of information not learnt. However, it is difficult to always obtain expected answers even with RAG, and one of the reasons for this is the inability to obtain chunks that contain correct answers to user questions or requests, that is, queries. We consider it important to quantitatively evaluate the effect of structure inside the obtained chunks. In this paper, we focus on the similarity between queries and chunks in RAG architecture. The experiments use three embedding models for validation, and the results show that some models tend to increase the cosine similarity when the relevant words of the query are located at the beginning of the chunks. On the other hand, no such tendency was observed when the related words were placed near the end of the chunk, suggesting that the location of the related words influences the accuracy of acquiring chunks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Similarity Evaluation Between Queries and Chunks in RAG Systems

  • Shun Noguchi,
  • Kozo Okano,
  • Shipei Ogata,
  • Shin Nakajima

摘要

While Large Language Models (LLMs) demonstrate excellent performance, their output depends on pre-trained knowledge. It has been pointed out that LLMs have various problems, such as generating outdated or incorrect information and they cannot always generate accurate answers. Retrieval-Augmented Generation (RAG) is a technical means that can mitigate the aforementioned problems by connecting an external database to LLM and retrieving relevant information from it, thereby making it possible to utilize a piece of information not learnt. However, it is difficult to always obtain expected answers even with RAG, and one of the reasons for this is the inability to obtain chunks that contain correct answers to user questions or requests, that is, queries. We consider it important to quantitatively evaluate the effect of structure inside the obtained chunks. In this paper, we focus on the similarity between queries and chunks in RAG architecture. The experiments use three embedding models for validation, and the results show that some models tend to increase the cosine similarity when the relevant words of the query are located at the beginning of the chunks. On the other hand, no such tendency was observed when the related words were placed near the end of the chunk, suggesting that the location of the related words influences the accuracy of acquiring chunks.