Audio-visual navigation (AVN) tasks require an embodied agent to locate sound sources in unfamiliar environments by integrating visual images and auditory cues. However, traditional multimodal fusion methods are limited in modeling deep collaborative relationships between vision and audio. To address this issue, this paper proposes a Dual-Stream Gated Fusion (DSGF) method. DSGF dynamically adjusts the fusion ratio between different modalities through a gating mechanism and introduces a residual structure to achieve multi-scale fusion and cross-modal feature enhancement. Experiments conducted on the Replica and Matterport3D datasets demonstrate that DSGF significantly outperforms existing methods in terms of navigation success rate and path efficiency, while also exhibiting greater robustness.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-Stream Gated Fusion for Audio-Visual Navigation

  • Jia Li,
  • Meiling Zhu,
  • Yinfeng Yu

摘要

Audio-visual navigation (AVN) tasks require an embodied agent to locate sound sources in unfamiliar environments by integrating visual images and auditory cues. However, traditional multimodal fusion methods are limited in modeling deep collaborative relationships between vision and audio. To address this issue, this paper proposes a Dual-Stream Gated Fusion (DSGF) method. DSGF dynamically adjusts the fusion ratio between different modalities through a gating mechanism and introduces a residual structure to achieve multi-scale fusion and cross-modal feature enhancement. Experiments conducted on the Replica and Matterport3D datasets demonstrate that DSGF significantly outperforms existing methods in terms of navigation success rate and path efficiency, while also exhibiting greater robustness.