Scholarlink

<p>Large language models are increasingly deployed on device and at the edge, where memory capacity, bandwidth, latency, and privacy requirements dominate system behavior. This survey systematizes the end side stack from algorithms to systems. On the model side, we present a clear taxonomy of quantization, pruning, knowledge distillation, low rank adaptation, and hybrid pipelines, explaining where representative methods belong and how they compose. On the system side, we link these techniques to inference frameworks, compiler and runtime optimizations, kernel fusion, and explicit management of the KV cache. We further propose a unified ALEM protocol, namely Accuracy, Latency, Energy, and Memory, and instantiate it on representative models from 1 to 4 billion parameters to reveal practical trade offs: apply quantization first for memory and time to first token, pair structured pruning with mergeable low rank compensation, and treat the KV cache as a first class subsystem through paging, compression, and eviction. Finally, we outline open problems and directions, including a unified low bit pipeline that couples transform, calibration, and kernel fusion, joint search over structured pruning and distillation, and train and serve unification that collapses sparse, quantized, and low rank parameters into inference ready weights. The goal is a practical bridge from algorithmic compression to resource aligned and reliable on device and edge deployment. <a href="https://github.com/LumosJiang/Awesome-On-Device-LLMs">https://github.com/LumosJiang/Awesome-On-Device-LLMs</a>: a repository hosting the complete references for Sects. 3–4.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

On-device large language models: a survey of model compression and system optimization

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月22日

查看全文
  • Wanyi Chen,
  • Junhao Wang,
  • Yiwei Zhang,
  • Yufan Shi,
  • Tianyi Jiang,
  • Shengxian Zhou,
  • Chenxu Wu,
  • Andi Zhang,
  • Chenyue Zhou,
  • Minxuan Wang,
  • Xinyu Liu,
  • Xiaoshuai Hao,
  • Yinan Wu,
  • Yichen Li,
  • Yuwei Hu,
  • Zhao Cao,
  • Yang Lu,
  • Mengke Li,
  • Yanbiao Ma,
  • Zhiwu Lu,
  • Jungong Han,
  • Yike Guo

摘要

Large language models are increasingly deployed on device and at the edge, where memory capacity, bandwidth, latency, and privacy requirements dominate system behavior. This survey systematizes the end side stack from algorithms to systems. On the model side, we present a clear taxonomy of quantization, pruning, knowledge distillation, low rank adaptation, and hybrid pipelines, explaining where representative methods belong and how they compose. On the system side, we link these techniques to inference frameworks, compiler and runtime optimizations, kernel fusion, and explicit management of the KV cache. We further propose a unified ALEM protocol, namely Accuracy, Latency, Energy, and Memory, and instantiate it on representative models from 1 to 4 billion parameters to reveal practical trade offs: apply quantization first for memory and time to first token, pair structured pruning with mergeable low rank compensation, and treat the KV cache as a first class subsystem through paging, compression, and eviction. Finally, we outline open problems and directions, including a unified low bit pipeline that couples transform, calibration, and kernel fusion, joint search over structured pruning and distillation, and train and serve unification that collapses sparse, quantized, and low rank parameters into inference ready weights. The goal is a practical bridge from algorithmic compression to resource aligned and reliable on device and edge deployment. https://github.com/LumosJiang/Awesome-On-Device-LLMs: a repository hosting the complete references for Sects. 3–4.