SMFNet: Streaming Multi-modal Audio-Visual Source Separation via Gated Fusion
摘要
Previous Audio-Visual Source Separation methods often rely on simple concatenation or element-wise multiplication for multi-modal fusion, limiting their ability to capture dynamic inter-modal relationships. With the inclusion of emerging modalities like motion, traditional single-stage fusion strategies face clear challenges. To overcome this, we propose a streaming multi-modal dynamic fusion approach based on a gating mechanism, which progressively integrates visual, auditory, and motion features. A visual attention module is also introduced to enhance visual representations. Experiments on the MUSIC and MUSIC21 datasets show that our method significantly outperforms existing approaches, demonstrating the effectiveness of the proposed gating and multi-stage fusion strategies.