<p>Audio-Visual Segmentation aims to pixel-wise locate and segment sounding objects in videos driven by audio cues. However, current mainstream methods typically employ audio-centric Transformer frameworks that derive object queries primarily from audio features. However, the modality gap limits these audio-centric approaches from accurately and consistently identifying sounds in videos, as relying on temporal audio signals to resolve spatial segmentation tasks may lead to perceptual ambiguity and a loss of fine-grained visual details, particularly in complex acoustic environments. To address these challenges, this paper proposes a novel visually-guided framework incorporating a Multi-Scale Fusion module and a Content-Guided Attention Fusion mechanism to priorities visual information to generate visually-derived queries, which then interact with audio context within a Transformer decoder for deep semantic refinement. Extensive experiments on standard benchmarks demonstrate that our approach achieves outperforming performance.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Visually-guided audio-visual aegmentation via multi-scale fusion and content-guided attention

  • Ying Cao,
  • Sikun Meng,
  • Yonghang Yan,
  • Hengyi Ren

摘要

Audio-Visual Segmentation aims to pixel-wise locate and segment sounding objects in videos driven by audio cues. However, current mainstream methods typically employ audio-centric Transformer frameworks that derive object queries primarily from audio features. However, the modality gap limits these audio-centric approaches from accurately and consistently identifying sounds in videos, as relying on temporal audio signals to resolve spatial segmentation tasks may lead to perceptual ambiguity and a loss of fine-grained visual details, particularly in complex acoustic environments. To address these challenges, this paper proposes a novel visually-guided framework incorporating a Multi-Scale Fusion module and a Content-Guided Attention Fusion mechanism to priorities visual information to generate visually-derived queries, which then interact with audio context within a Transformer decoder for deep semantic refinement. Extensive experiments on standard benchmarks demonstrate that our approach achieves outperforming performance.