Slime: An Intelligent Hybrid Recovery Strategy that Minimizes Recovery Overhead
摘要
The remarkable generalization capabilities of large language models (LLMs), along with their expert-level performance across various domains, have positioned them as a key driver of the next technological revolution, leading to their widespread deployment. However, the presence of the Key-Value(KV) cache introduces significant challenges in efficient memory management, which is crucial for deploying LLMs effectively.To address this, dynamic memory management solutions such as PagedAttention have been proposed to allocate GPU memory on demand. However, to mitigate the risk of out-of-memory (OOM) errors, some requests must be suspended and later reactivated when sufficient memory becomes available. This reactivation process, however, incurs substantial recovery latency, leading to inference delays. To reduce recovery latency, we propose Slime, an adaptive hybrid recovery strategy that dynamically partitions recovery tasks into Recomputation and Load operations based on predictor-guided decisions. By leveraging the computation-communication overlap of GPUs, Slime significantly minimizes recovery latency. Experimental results show that Slime reduces recovery latency by approximately 40% compared to the best-performing single recovery strategy, while consistently selecting the optimal recovery approach across all scenarios.