Scholarlink

This paper presents a novel query-specific knowledge cache system, RAG-Cache, to accelerate retrieval-augmented generation (RAG). The novelty of RAG-Cache is two-fold: (1) it proposes a memory-efficient non-redundant three-layer cache structure for caching knowledge for RAG; (2) it presents a new RAG cache replacement policy, HARF (Hotness-Aware Rechecked FIFO), which combines the second-chance FIFO and a space-efficient 2-bit reference count to reflect both temporal and hotness of queries, offering high concurrency performance. After introducing the RAG-Cache’s architecture, we detail the cache structure and the HARF policy. Finally, we present the evaluation results and the demonstration design. The results show that RAG-Cache reduces average TTFT by 67% and 26%, and increases average throughput by \(1.99\times \) and 34%, compared with RAG systems without caching and with FIFO caching.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

RAG-Cache: Efficient Query-Specific Knowledge Caching for Retrieval Augmented Generation

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Haoying Jin,
  • Haoyang Feng

摘要

This paper presents a novel query-specific knowledge cache system, RAG-Cache, to accelerate retrieval-augmented generation (RAG). The novelty of RAG-Cache is two-fold: (1) it proposes a memory-efficient non-redundant three-layer cache structure for caching knowledge for RAG; (2) it presents a new RAG cache replacement policy, HARF (Hotness-Aware Rechecked FIFO), which combines the second-chance FIFO and a space-efficient 2-bit reference count to reflect both temporal and hotness of queries, offering high concurrency performance. After introducing the RAG-Cache’s architecture, we detail the cache structure and the HARF policy. Finally, we present the evaluation results and the demonstration design. The results show that RAG-Cache reduces average TTFT by 67% and 26%, and increases average throughput by \(1.99\times \) and 34%, compared with RAG systems without caching and with FIFO caching.