Scholarlink

<p>Fine-tuning Visual Foundation Models (VFMs) is proven to be effective in Domain Generalization Semantic Segmentation (DGSS). The state space model has become a candidate for lightweight fine-tuning due to its advantages in long-range modeling and dynamic feature adaptation. However, how to guide the state-space model to learn domain-common features and how to adapt it to the multi-level feature system of VFMs remain to be solved. In this paper, we propose a state space based parameters-efficient fine-tuning method, SLFMamba which introduces state space into model fine-tuning through a set of trainable tokens. First, pixel-level embedding is performed on each layer of features of VFMs. Second, the feature map is dot-multiplied with the trainable tokens and then passed through the Selective Scan Module (SSM) to obtain an attention-like similarity map. This similarity map can better refine each instance of the image. Finally, to reduce the complexity of the model, the trainable tokens are decomposed into two low-rank matrices <i>A</i> and <i>B</i>, and the parameters of the matrix <i>B</i> are shared, namely Shared LoRA (SLoRA). At the same time, the MLP and SSM of each layer are shared. In the experiments with different settings of DGSS, the effect of SLFMamba is better than other models. Remarkably, using only <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(0.9\%\)</EquationSource> </InlineEquation> additional trainable parameters in the frozen backbone, SLFMamba achieves <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(68.1\%\)</EquationSource> </InlineEquation> mIoU on Cityscapes without access to any real-world data. It turns out that state space has great potential for model fine-tuning. The code is available in <a href="https://github.com/xiaoxia0722/SLFMamba">https://github.com/xiaoxia0722/SLFMamba</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Slfmamba:a state space based vision foundation models fine-tuning for domain generalized semantic segmentations

  • 期刊论文

  • 发表日期：

    2025年10月31日

查看全文
  • Yongchao Qiao,
  • Ya’nan Guan,
  • Qihan He,
  • Zhongxu Li,
  • Jingmin Yang,
  • Wenyuan Yang

摘要

Fine-tuning Visual Foundation Models (VFMs) is proven to be effective in Domain Generalization Semantic Segmentation (DGSS). The state space model has become a candidate for lightweight fine-tuning due to its advantages in long-range modeling and dynamic feature adaptation. However, how to guide the state-space model to learn domain-common features and how to adapt it to the multi-level feature system of VFMs remain to be solved. In this paper, we propose a state space based parameters-efficient fine-tuning method, SLFMamba which introduces state space into model fine-tuning through a set of trainable tokens. First, pixel-level embedding is performed on each layer of features of VFMs. Second, the feature map is dot-multiplied with the trainable tokens and then passed through the Selective Scan Module (SSM) to obtain an attention-like similarity map. This similarity map can better refine each instance of the image. Finally, to reduce the complexity of the model, the trainable tokens are decomposed into two low-rank matrices A and B, and the parameters of the matrix B are shared, namely Shared LoRA (SLoRA). At the same time, the MLP and SSM of each layer are shared. In the experiments with different settings of DGSS, the effect of SLFMamba is better than other models. Remarkably, using only \(0.9\%\) additional trainable parameters in the frozen backbone, SLFMamba achieves \(68.1\%\) mIoU on Cityscapes without access to any real-world data. It turns out that state space has great potential for model fine-tuning. The code is available in https://github.com/xiaoxia0722/SLFMamba.