Scholarlink

<p>This research proposes a hybrid approach that combines linear attention, chunking, and recurrent mechanisms to address the efficiency issues of Large Language Models(LLMs) within the traditional transformer framework. Our approach integrates three key innovations: We use linear attention to employ kernel function mapping to reduce time and space complexity from <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(O(n^2)\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>O</mi> <mo stretchy="false">(</mo> <msup> <mi>n</mi> <mn>2</mn> </msup> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation> to <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(O(n)\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>O</mi> <mo stretchy="false">(</mo> <mi>n</mi> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation>; The proposed dynamic chunk-based processing, can compress 5 times KV cache with mean pooling; Through 3 different ways, our hard thresholding, adaptive gating, and hierarchical chunking, can filter token and reduce load. The result shows that it can actually improve the efficiency of LLM, and performs excellently among some evaluation tools. Experiments demonstrate that our 3.2B parameter model achieves excellent performance in multiple benchmark tests, outperforming dense models of similar scale and even matching the performance of larger models in certain tasks, which provides a theoretically grounded and empirically validated framework for efficient LLM optimization.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Efficient optimization of large language models: a hybrid approach combining linear attention, chunk, and recurrent

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月9日

查看全文
  • Cheng Zhang,
  • Linlin Shen,
  • Yudong Li

摘要

This research proposes a hybrid approach that combines linear attention, chunking, and recurrent mechanisms to address the efficiency issues of Large Language Models(LLMs) within the traditional transformer framework. Our approach integrates three key innovations: We use linear attention to employ kernel function mapping to reduce time and space complexity from \(O(n^2)\) O ( n 2 ) to \(O(n)\) O ( n ) ; The proposed dynamic chunk-based processing, can compress 5 times KV cache with mean pooling; Through 3 different ways, our hard thresholding, adaptive gating, and hierarchical chunking, can filter token and reduce load. The result shows that it can actually improve the efficiency of LLM, and performs excellently among some evaluation tools. Experiments demonstrate that our 3.2B parameter model achieves excellent performance in multiple benchmark tests, outperforming dense models of similar scale and even matching the performance of larger models in certain tasks, which provides a theoretically grounded and empirically validated framework for efficient LLM optimization.