Scholarlink

In the realm of Visual Question Answering (VQA), the integration of image understanding and language processing through models like the Bootstrapping Language-Image Pre-training (BLIP) has significantly advanced the field. However, as Multimodal Models (MM) grow in size, adapting such comprehensive models to low-resource VQA tasks becomes increasingly challenging due to high computational costs and storage requirements. To address these challenges, we introduce AdaQLoRA, which adaptively adjusts the allocation of parameters across weight matrices, leveraging quantization techniques and Singular Value Decomposition (SVD) enhanced Low-rank Adapters (LoRA). This approach not only prioritizes weight matrices crucial for performance enhancement but also achieves a balance between low-precision storage and high-precision computation. Our strategy markedly improves computational efficiency for key parameters and reduces memory usage, thereby enhancing parameter utilization and the performance of the fine-tuned model. Extensive experiments on datasets such as VQA2.0, VizWiz, and Visual7W demonstrate that AdaQLoRA surpasses existing parameter-efficient tuning methods in terms of model performance, parameter efficiency, and training cost reduction.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

AdaQLoRA: A Novel Paradigm in Low-Resource Visual Question Answering with Adaptive Quantization and Low-Rank Optimization

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Peng Yang,
  • Hong Ying,
  • Jianxin Duan,
  • Manman Lin,
  • Wenli Bai

摘要

In the realm of Visual Question Answering (VQA), the integration of image understanding and language processing through models like the Bootstrapping Language-Image Pre-training (BLIP) has significantly advanced the field. However, as Multimodal Models (MM) grow in size, adapting such comprehensive models to low-resource VQA tasks becomes increasingly challenging due to high computational costs and storage requirements. To address these challenges, we introduce AdaQLoRA, which adaptively adjusts the allocation of parameters across weight matrices, leveraging quantization techniques and Singular Value Decomposition (SVD) enhanced Low-rank Adapters (LoRA). This approach not only prioritizes weight matrices crucial for performance enhancement but also achieves a balance between low-precision storage and high-precision computation. Our strategy markedly improves computational efficiency for key parameters and reduces memory usage, thereby enhancing parameter utilization and the performance of the fine-tuned model. Extensive experiments on datasets such as VQA2.0, VizWiz, and Visual7W demonstrate that AdaQLoRA surpasses existing parameter-efficient tuning methods in terms of model performance, parameter efficiency, and training cost reduction.