We present an approach to audio-visual speaker recognition that consists of two parts. First, we extend an audio-only ECAPA-TDNN model with an audio-visual (AV) fusion module that fuses both modalities to process the results in the ECAPA model. Second, we extend a ResNet that processes individual frames for face recognition with an attention block that processes a sequence of embeddings to focus on meaningful embeddings and thus increase the robustness towards quality differences in images. The fusion of the individual model’s results further improves the performance and reduces the error rates. Comparing our models to the baselines reveals that the additional visual information and attention-based processing of image sequences result in relative improvements of up to 56%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Combining Temporal Visual Dynamics and Audio Representations for Robust Speaker Identification

  • Christopher Simic,
  • Korbinian Riedhammer,
  • Tobias Bocklet

摘要

We present an approach to audio-visual speaker recognition that consists of two parts. First, we extend an audio-only ECAPA-TDNN model with an audio-visual (AV) fusion module that fuses both modalities to process the results in the ECAPA model. Second, we extend a ResNet that processes individual frames for face recognition with an attention block that processes a sequence of embeddings to focus on meaningful embeddings and thus increase the robustness towards quality differences in images. The fusion of the individual model’s results further improves the performance and reduces the error rates. Comparing our models to the baselines reveals that the additional visual information and attention-based processing of image sequences result in relative improvements of up to 56%.