Leveraging CRNN and EfficientNet-B0 for Deepfake Detection in the FakeAVCeleb Dataset
摘要
The presence of deep learning technologies, in recent years, has triggered the emergence of deepfakes artificial media wherein the face of a human on an existing image, video, or audio is replaced with the face of the other person. The implications of such a phenomenon are devastating with regard to privacy, security, and regulation of misinformation. The study proposes a new multimodal deepfake detection model that integrates both visual and audio features to deliver competent and efficient results. Visual feature detection in the form of the EfficientNet-B0 model is applied to extract facial frames and a Convolutional Recurrent Neural Network (CRNN) is applied to put the audio features in the form of mel-spectrograms. A weighted late fusion method combines modality-specific predictions, which makes use of each of their strengths. The approach is tested on the FakeAVCeleb- the dataset composed of labeled pairs of real and fake audio-visual examples-and beats the unimodal and state-of-the-art baselines over both accuracy and F1-scores by achieving 99.82 and 99.78% respectively in identifying fake classes. The results establish directions of future research on lightweight and powerful architecture-based multimodal misinformation detection.