<p>Audio-Visual Segmentation (AVS) is a task aimed at predicting pixel-level masks for sound-producing objects in videos. Existing AVS methods typically involve a heavy audio encoder and use full fine-tuning or additional structures like adapters on the visual branch. We argue that these approaches incur significant training costs and may disrupt the pretrained model’s prior knowledge. To address this, we propose a novel hierarchical side-tuning framework for AVS, utilizing a side network that simultaneously performs audio encoding and cross-domain adaptation. By freezing the visual encoder and only tuning the side network, we significantly reduce the number of parameters to be trained. Additionally, inspired by the recent success of state space models (SSMs), we introduce an Audio-Visual Fusion (AVF) module in the side network and design a lightweight SSM-based decoder to enhance feature fusion and decoding. Experimental results on multiple AVS datasets demonstrate that our method achieves competitive or even superior performance compared with state-of-the-art approaches, while using fewer learnable parameters.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Audio–visual segmentation via hierarchical side tuning with state space model

  • Wenyi Xia,
  • Qingwei Geng,
  • Xiaodong Gu

摘要

Audio-Visual Segmentation (AVS) is a task aimed at predicting pixel-level masks for sound-producing objects in videos. Existing AVS methods typically involve a heavy audio encoder and use full fine-tuning or additional structures like adapters on the visual branch. We argue that these approaches incur significant training costs and may disrupt the pretrained model’s prior knowledge. To address this, we propose a novel hierarchical side-tuning framework for AVS, utilizing a side network that simultaneously performs audio encoding and cross-domain adaptation. By freezing the visual encoder and only tuning the side network, we significantly reduce the number of parameters to be trained. Additionally, inspired by the recent success of state space models (SSMs), we introduce an Audio-Visual Fusion (AVF) module in the side network and design a lightweight SSM-based decoder to enhance feature fusion and decoding. Experimental results on multiple AVS datasets demonstrate that our method achieves competitive or even superior performance compared with state-of-the-art approaches, while using fewer learnable parameters.