Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing and other complex tasks. However, multi-agent deployments of LLMs often suffer from significant inference overhead caused by a great deal of context and repeated tokens across agents, resulting in high Time-To-First-Token (TTFT) and elevated computational costs. To address these challenges, we propose EACC (Efficient Agent Context Cache), a system that enables position-independent reuse of key-value (KV) caches across agents by semantically chunking the prompt and managing reusable KV pairs through a globally accessible Agent Context Cache. EACC decouples context storage from computation, supports distributed deployment, and integrates with existing Position-Independent Caching (PIC) techniques such as CacheBlend and EPIC for selective recomputation. Our evaluation across multiple datasets and LLM backends demonstrates that EACC achieves up to 82.4% reduction in TTFT, while maintaining output accuracy, thereby offering a scalable and efficient inference infrastructure for multi-agent services.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EACC: Efficient Agent Context Cache Sharing for Multi-Agent Systems

  • Sihao Cheng,
  • Yunfei Gu,
  • Chentao Wu,
  • Jie Meng

摘要

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing and other complex tasks. However, multi-agent deployments of LLMs often suffer from significant inference overhead caused by a great deal of context and repeated tokens across agents, resulting in high Time-To-First-Token (TTFT) and elevated computational costs. To address these challenges, we propose EACC (Efficient Agent Context Cache), a system that enables position-independent reuse of key-value (KV) caches across agents by semantically chunking the prompt and managing reusable KV pairs through a globally accessible Agent Context Cache. EACC decouples context storage from computation, supports distributed deployment, and integrates with existing Position-Independent Caching (PIC) techniques such as CacheBlend and EPIC for selective recomputation. Our evaluation across multiple datasets and LLM backends demonstrates that EACC achieves up to 82.4% reduction in TTFT, while maintaining output accuracy, thereby offering a scalable and efficient inference infrastructure for multi-agent services.