Scholarlink

Abstract <p>Large Language Models (LLMs) require a lot of computational resources for inference. That is why the latest advancements in hardware design may offer many possibilities for speeding the LLM up. For example, TPU optimize calculations on data, transformed into the Coordinate sparse tensor format. The SparseCore processing unit that performs the calculations is heavily tailored for the extremely sparse embeddings of Deep Learning Recommendation Models. The other example of the enhanced hardware is Sparse Tensor Cores, that offer support for <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(n:m\)</EquationSource> <!--OptMem2560179Demidovskii-m1--> </InlineEquation> data structure (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(n\)</EquationSource> <!--OptMem2560179Demidovskii-m2--> </InlineEquation> zeroes out of every subsequent <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(m\)</EquationSource> <!--OptMem2560179Demidovskii-m3--> </InlineEquation> elements), that allows to drastically reduce the calculations by compressing the original matrix into a dense one. Methods like Wanda and SliceGPT prepare LLM weights to harness the power of the latter. However, as the weights are the most crucial assets of any model, it appears to be a good idea to modify the activations instead. This article introduces a novel dynamic sparsification algorithm called KurSparse , which proposes fine-grained <i>n</i> : <i>m</i> sparsity pattern, that affects only a portion of channels. This portion is selected with kurtosis threshold <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(\zeta \)</EquationSource> <!--OptMem2560179Demidovskii-m4--> </InlineEquation>. The proposed method shows significant reduction in MAC operations by 3.1x with average quality drop for LLaMA-3.1-8B model less than 2%.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Novel Activation Sparsification Approach for Large Language Models

  • 期刊论文

  • 发表日期：

    2025年12月19日

查看全文
  • A. V. Demidovskij,
  • E. O. Burmistrova,
  • E. I. Zharikov

摘要

Abstract

Large Language Models (LLMs) require a lot of computational resources for inference. That is why the latest advancements in hardware design may offer many possibilities for speeding the LLM up. For example, TPU optimize calculations on data, transformed into the Coordinate sparse tensor format. The SparseCore processing unit that performs the calculations is heavily tailored for the extremely sparse embeddings of Deep Learning Recommendation Models. The other example of the enhanced hardware is Sparse Tensor Cores, that offer support for \(n:m\) data structure ( \(n\) zeroes out of every subsequent \(m\) elements), that allows to drastically reduce the calculations by compressing the original matrix into a dense one. Methods like Wanda and SliceGPT prepare LLM weights to harness the power of the latter. However, as the weights are the most crucial assets of any model, it appears to be a good idea to modify the activations instead. This article introduces a novel dynamic sparsification algorithm called KurSparse , which proposes fine-grained n : m sparsity pattern, that affects only a portion of channels. This portion is selected with kurtosis threshold \(\zeta \) . The proposed method shows significant reduction in MAC operations by 3.1x with average quality drop for LLaMA-3.1-8B model less than 2%.