Audio-visual embodied navigation relies on sound and visual data from the environment to complete navigation tasks in a 3D environment. During its propagation, sound is influenced by various factors, such as the environment. However, it plays a crucial role in perceiving and recognizing objects. In this paper, we propose enhancing sound features with visual cues to disentangle acoustic ambiguities. By using audio with enhanced sound features for audio-visual navigation tasks, we can improve navigation efficiency. However, our experiments revealed that simply combining audio with extracted sound features and visual depth maps or RGB images does not significantly enhance navigation efficiency. Therefore, we propose using feature decoupling to process sound and vision separately. This approach improves the efficiency of audio-visual navigation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Disentangling Acoustic Ambiguities with Visual Cues for Audio-Visual Navigation

  • Shaohang Wu,
  • Meiling Zhu,
  • Yinfeng Yu

摘要

Audio-visual embodied navigation relies on sound and visual data from the environment to complete navigation tasks in a 3D environment. During its propagation, sound is influenced by various factors, such as the environment. However, it plays a crucial role in perceiving and recognizing objects. In this paper, we propose enhancing sound features with visual cues to disentangle acoustic ambiguities. By using audio with enhanced sound features for audio-visual navigation tasks, we can improve navigation efficiency. However, our experiments revealed that simply combining audio with extracted sound features and visual depth maps or RGB images does not significantly enhance navigation efficiency. Therefore, we propose using feature decoupling to process sound and vision separately. This approach improves the efficiency of audio-visual navigation.