For consumer usage of locally deployed LLMs, the GGUF format and k_quantization are invaluable tools for maintaining the performance of the original model while reducing it to sizes deployable with consumer-grade hardware. This size reduction is achieved by reducing the number of bits dedicated to each weight in the original model based on how important they are thought to be during model inference. A weight’s importance is arrived at through the application of an ‘importance matrix’—a relatively small text document meant to be representative of the LLM’s standard use-cases. For the vast majority of quantized LLMs available online, this document is primarily written in English. It was therefore an open question whether performance on English language tasks was preserved through the sacrifice of multilingual performance and whether it can be preserved with alternate importance matrices. This article investigates these hypotheses by quantizing Llama3.3 70B on importance matrices written in three languages (English, Norwegian, and Malayalam) and evaluating them on the MixEval dataset in both English and Norwegian. All experiments yielded nonsignificant results, indicating that current quantization practices do not disproportionately harm multilingual performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance

  • Karl Audun Kagnes Borgersen,
  • Morten Goodwin

摘要

For consumer usage of locally deployed LLMs, the GGUF format and k_quantization are invaluable tools for maintaining the performance of the original model while reducing it to sizes deployable with consumer-grade hardware. This size reduction is achieved by reducing the number of bits dedicated to each weight in the original model based on how important they are thought to be during model inference. A weight’s importance is arrived at through the application of an ‘importance matrix’—a relatively small text document meant to be representative of the LLM’s standard use-cases. For the vast majority of quantized LLMs available online, this document is primarily written in English. It was therefore an open question whether performance on English language tasks was preserved through the sacrifice of multilingual performance and whether it can be preserved with alternate importance matrices. This article investigates these hypotheses by quantizing Llama3.3 70B on importance matrices written in three languages (English, Norwegian, and Malayalam) and evaluating them on the MixEval dataset in both English and Norwegian. All experiments yielded nonsignificant results, indicating that current quantization practices do not disproportionately harm multilingual performance.