Retrieval-augmented generation (RAG) tools aim to combine language models and real-time database search to mitigate AI hallucinations. This study, conducted in the fall of 2024, assessed four academic RAG tools with respect to their ease of use as well as relevance of search results. The evaluation included a survey and a series of workshops with researchers at KTH Royal Institute of Technology, as well as additional tests conducted by library staff following a structured search protocol. Our findings suggest that, at this stage, these tools cannot replace traditional systematic database searches. A key weakness of these tools is the lack of context provided with the search results. We argue, in conclusion, for establishing some guiding principles for appropriate use of RAG tools in different use cases.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

RAG in Research: Evaluating AI-Driven Literature Search Tools

  • Pontus Juth,
  • Magdalena Svanberg,
  • Mattias Vesterlund,
  • Agne Larsson,
  • Ika Jorum,
  • Sofie Seo,
  • Annika Peurell,
  • Elin Palm

摘要

Retrieval-augmented generation (RAG) tools aim to combine language models and real-time database search to mitigate AI hallucinations. This study, conducted in the fall of 2024, assessed four academic RAG tools with respect to their ease of use as well as relevance of search results. The evaluation included a survey and a series of workshops with researchers at KTH Royal Institute of Technology, as well as additional tests conducted by library staff following a structured search protocol. Our findings suggest that, at this stage, these tools cannot replace traditional systematic database searches. A key weakness of these tools is the lack of context provided with the search results. We argue, in conclusion, for establishing some guiding principles for appropriate use of RAG tools in different use cases.