The emergence of multimodal deepfake videos demands detection systems that address synchronized audio-visual manipulations. We present AVMCD, a novel framework combining dual-stream transformer architecture with cross-modal consistency verification. Our solution introduces three key innovations: (1) Joint spatiotemporal modeling using Video Vision Transformers (ViViT) for facial dynamics and Audio Spectrogram Transformers (AST) for spectral speech patterns; (2) A synchronization analysis module employing cross-module features to detect audio-visual temporal mismatches; (3) Hybrid learning integrating one-class classification with multi-task consistency constraints. The framework overcomes critical limitations in existing approaches by explicitly modeling cross-modal interactions while preventing overfitting to single-modality artifacts. Comprehensive evaluations on FakeAVCeleb demonstrate state-of-the-art performance with 96.1% ACC, surpassing leading audio-visual methods by 12.4% ACC improvement. This work establishes a new paradigm for multimodal deepfake detection through systematic integration of transformer-based feature fusion and physiological consistency verification.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Consistency-Driven Deepfake Detection

  • Li Zhang,
  • Bin Liu,
  • Qi Chu,
  • Nenghai Yu

摘要

The emergence of multimodal deepfake videos demands detection systems that address synchronized audio-visual manipulations. We present AVMCD, a novel framework combining dual-stream transformer architecture with cross-modal consistency verification. Our solution introduces three key innovations: (1) Joint spatiotemporal modeling using Video Vision Transformers (ViViT) for facial dynamics and Audio Spectrogram Transformers (AST) for spectral speech patterns; (2) A synchronization analysis module employing cross-module features to detect audio-visual temporal mismatches; (3) Hybrid learning integrating one-class classification with multi-task consistency constraints. The framework overcomes critical limitations in existing approaches by explicitly modeling cross-modal interactions while preventing overfitting to single-modality artifacts. Comprehensive evaluations on FakeAVCeleb demonstrate state-of-the-art performance with 96.1% ACC, surpassing leading audio-visual methods by 12.4% ACC improvement. This work establishes a new paradigm for multimodal deepfake detection through systematic integration of transformer-based feature fusion and physiological consistency verification.