Scholarlink

<p>The checkpoint of large language models (LLMs) typically range from tens of gigabytes to terabytes in storage size, with loading processes often requiring several seconds to minutes. This substantial latency issue significantly impacts service level objective compliance for providers of large-scale model services. To address the LLM loading cold-start problem, we present LLM FlowLoader, a model loader featuring multi-tier local caching and parallel communication pipeline optimization. By strategically leveraging GPU servers’ heterogeneous network resources and storage hierarchies, our system accelerates checkpoint loading through two key innovations: (1) An asynchronous pipeline transporter based on task stealing and affinity domain awareness, which transforms complex network transmission into simplified chain-structured transmission by introducing affinity domains, and adaptively enhances I/O bandwidth through a task-stealing algorithm. (2) A multi-tier caching system co-designed with the transporter that exploits GPU servers’ multi-tier memory/storage architecture to accelerate data accessibility and reduce cold-start latency. Experimental results demonstrate that LLM FlowLoader not only achieves significant loading speed improvements but also provides novel insights for optimizing GPU resource utilization.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

FlowLoader: an efficient large language model loader via parallel transmissions and local multi-tier caching

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年8月12日

查看全文
  • Jun Yin,
  • Fei Wu,
  • Shihao Li,
  • Peng Huang

摘要

The checkpoint of large language models (LLMs) typically range from tens of gigabytes to terabytes in storage size, with loading processes often requiring several seconds to minutes. This substantial latency issue significantly impacts service level objective compliance for providers of large-scale model services. To address the LLM loading cold-start problem, we present LLM FlowLoader, a model loader featuring multi-tier local caching and parallel communication pipeline optimization. By strategically leveraging GPU servers’ heterogeneous network resources and storage hierarchies, our system accelerates checkpoint loading through two key innovations: (1) An asynchronous pipeline transporter based on task stealing and affinity domain awareness, which transforms complex network transmission into simplified chain-structured transmission by introducing affinity domains, and adaptively enhances I/O bandwidth through a task-stealing algorithm. (2) A multi-tier caching system co-designed with the transporter that exploits GPU servers’ multi-tier memory/storage architecture to accelerate data accessibility and reduce cold-start latency. Experimental results demonstrate that LLM FlowLoader not only achieves significant loading speed improvements but also provides novel insights for optimizing GPU resource utilization.