Scholarlink

We propose QDLoRA, a parameter-efficient fine-tuning (PEFT) framework that integrates low-rank decomposition and quantization into the LoRA fine-tuning process for pretrained large language models (LLMs). Unlike prior methods such as LoftQ and ApiQ that rely solely on quantization and suffer performance degradation under extreme compression, QDLoRA preserves more informative structure at the same compression ratio, thereby improving fine-tuning results. To further enhance robustness, QDLoRA introduces a similarity-aware rank selection strategy and a quantization-aware initialization scheme. Experimental results on various model architectures across diverse NLP benchmarks demonstrate that QDLoRA achieves superior accuracy and efficiency compared to existing methods, particularly under limited resource budgets. The proposed method offers a practical and scalable solution for efficient fine-tuning of large language models.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

QDLoRA: Enhanced LoRA Fine-Tuning on Quantized LLMs via Integrated Low-Rank Decomposition

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Xingyi Su,
  • Rui Wang,
  • Zhongzhi Luan,
  • Yi Liu,
  • Depei Qian

摘要

We propose QDLoRA, a parameter-efficient fine-tuning (PEFT) framework that integrates low-rank decomposition and quantization into the LoRA fine-tuning process for pretrained large language models (LLMs). Unlike prior methods such as LoftQ and ApiQ that rely solely on quantization and suffer performance degradation under extreme compression, QDLoRA preserves more informative structure at the same compression ratio, thereby improving fine-tuning results. To further enhance robustness, QDLoRA introduces a similarity-aware rank selection strategy and a quantization-aware initialization scheme. Experimental results on various model architectures across diverse NLP benchmarks demonstrate that QDLoRA achieves superior accuracy and efficiency compared to existing methods, particularly under limited resource budgets. The proposed method offers a practical and scalable solution for efficient fine-tuning of large language models.