Scholarlink

<p>The scale of large language models (LLMs) continues to grow in response to increasing demands for intelligent applications. When these large models and their intermediate results, such as key-value (KV) caches, are deployed in resource-constrained environments like edge inference scenarios, they impose substantial pressure on computational and storage resources, resulting in significant performance degradation and storage inefficiency. To address the problem, this paper proposes a novel Multi-Tier Dynamic Storage (MTDS) framework that offloads KV caches from limited GPU VRAM to a hierarchical storage system, effectively reducing both memory and computation overhead on the GPU. By introducing a selective KV cache reuse mechanism, MTDS achieves notable improvements in inference performance. We further develop a dynamic storage access control scheme and an adaptive hierarchical eviction strategy to address the challenges of bandwidth contention and capacity overhead introduced by multi-tier storage under limited resources. These techniques significantly alleviate performance bottlenecks and reduce resource waste in edge inference servers. Experimental results demonstrate that MTDS improves LLM inference efficiency, reduces first-token latency by more than 25%, and increases multi-tier active storage cache hit rate by up to 20%.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-tier dynamic storage of KV cache for LLM inference under resource-constrained conditions

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月27日

查看全文
  • Junliang Wang,
  • Jiaqi Hu,
  • Qingping Cao,
  • Yuanrui Zhu,
  • Xiancheng Lin

摘要

The scale of large language models (LLMs) continues to grow in response to increasing demands for intelligent applications. When these large models and their intermediate results, such as key-value (KV) caches, are deployed in resource-constrained environments like edge inference scenarios, they impose substantial pressure on computational and storage resources, resulting in significant performance degradation and storage inefficiency. To address the problem, this paper proposes a novel Multi-Tier Dynamic Storage (MTDS) framework that offloads KV caches from limited GPU VRAM to a hierarchical storage system, effectively reducing both memory and computation overhead on the GPU. By introducing a selective KV cache reuse mechanism, MTDS achieves notable improvements in inference performance. We further develop a dynamic storage access control scheme and an adaptive hierarchical eviction strategy to address the challenges of bandwidth contention and capacity overhead introduced by multi-tier storage under limited resources. These techniques significantly alleviate performance bottlenecks and reduce resource waste in edge inference servers. Experimental results demonstrate that MTDS improves LLM inference efficiency, reduces first-token latency by more than 25%, and increases multi-tier active storage cache hit rate by up to 20%.