Scholarlink

<p>Attention-based vision models, such as Vision Transformers, have garnered significant attention due to their effectiveness in various computer vision tasks. However, their large model sizes pose challenges for deployment on resource-constrained devices. Low-rank decomposition emerges as an effective technique to alleviate this issue by compressing models while preserving accuracy. The selection of the optimal rank value is crucial but often requires extensive neural architecture search (NAS), which can be computationally intensive. In this paper, we propose an adaptive ComCAT method that dynamically adjusts the frequency of NAS during low-rank decomposition by monitoring the differences in search results. By reducing unnecessary NAS iterations, we achieve faster training while maintaining model accuracy. Our experiments on pretrained DeiT-Small and DeiT-Base models demonstrate that our approach reduces training time by 13.86% and 40.77%, respectively, without compromising on Top-1 test set accuracy. This work highlights the potential of adaptive NAS strategies in optimizing low-rank decomposition for efficient vision models, and our code is available at <a href="https://github.com/YaoZhao11/Adaptive-ComCAT">https://github.com/YaoZhao11/Adaptive-ComCAT</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Optimizing low-rank decomposition for efficient attention-based vision models via adaptive neural architecture search

  • 期刊论文

  • 发表日期：

    2025年12月11日

查看全文
  • Yao Zhao,
  • Wei Lu,
  • Yinghui Zhang

摘要

Attention-based vision models, such as Vision Transformers, have garnered significant attention due to their effectiveness in various computer vision tasks. However, their large model sizes pose challenges for deployment on resource-constrained devices. Low-rank decomposition emerges as an effective technique to alleviate this issue by compressing models while preserving accuracy. The selection of the optimal rank value is crucial but often requires extensive neural architecture search (NAS), which can be computationally intensive. In this paper, we propose an adaptive ComCAT method that dynamically adjusts the frequency of NAS during low-rank decomposition by monitoring the differences in search results. By reducing unnecessary NAS iterations, we achieve faster training while maintaining model accuracy. Our experiments on pretrained DeiT-Small and DeiT-Base models demonstrate that our approach reduces training time by 13.86% and 40.77%, respectively, without compromising on Top-1 test set accuracy. This work highlights the potential of adaptive NAS strategies in optimizing low-rank decomposition for efficient vision models, and our code is available at https://github.com/YaoZhao11/Adaptive-ComCAT.