Scholarlink

<p>Large language models (LLMs) show impressive capabilities across many NLP tasks, but their enormous size creates major deployment challenges. While single compression methods provide limited solutions, combining approaches such as pruning, quantization, knowledge distillation, and low-rank approximation might be essential for both higher compression rates and better model performance.</p><p>This paper studies the synergistic effects of combining multiple LLM compression techniques. Our findings reveal that strategic combinations can potentially reduce model size by more than 90% while maintaining performance, with contextual pruning and quantization. Meanwhile, the order of application could impact outcomes, and that joint optimization of compression methods could outperform sequential combination. Although promising, existing combination approaches rely on manual design choices and lack a systematic framework for multi-technique compression. To address this, we prototype a formal framework for automated, multi-technique LLM compression that optimizes the combination sequence. Finally, we discuss remaining challenges and outline future research directions for more efficient large language models.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A study and formal framework of the composability of LLM compression techniques

  • 期刊论文

  • 发表日期：

    2026年2月12日

查看全文
  • Gansen Hu,
  • Zhaoguo Wang

摘要

Large language models (LLMs) show impressive capabilities across many NLP tasks, but their enormous size creates major deployment challenges. While single compression methods provide limited solutions, combining approaches such as pruning, quantization, knowledge distillation, and low-rank approximation might be essential for both higher compression rates and better model performance.

This paper studies the synergistic effects of combining multiple LLM compression techniques. Our findings reveal that strategic combinations can potentially reduce model size by more than 90% while maintaining performance, with contextual pruning and quantization. Meanwhile, the order of application could impact outcomes, and that joint optimization of compression methods could outperform sequential combination. Although promising, existing combination approaches rely on manual design choices and lack a systematic framework for multi-technique compression. To address this, we prototype a formal framework for automated, multi-technique LLM compression that optimizes the combination sequence. Finally, we discuss remaining challenges and outline future research directions for more efficient large language models.