The rapid advancement of deep learning models that can synthesize and generate hyper-realistic videos, known as Deepfakes, and their public accessibility have raised concerns about potential malicious uses. Deep learning techniques can now generate new faces, swap faces between two people in a photo or a video, alter facial expressions and facial features. These powerful video and photo manipulation methods can find potential use in many fields. However, they also pose a growing threat to everyone if individuals use them for harmful purposes such as identity theft, phishing, and scams. In this work, we propose a Vision Transformer and CNN hybrid for detecting fake faces. The hybrid model consists of two components: a Convolutional Neural Network (CNN) and a Vision Transformer (ViT). The CNN extracts local features, while the ViT captures and analyzes the global features that CNN might have missed. We trained our model on a combination of 140k Real and Fake Faces and Real and Fake Face Detection datasets and achieved 92.33% accuracy. Our contribution is that we added CNN for local feature capturing to the ViT architecture and got a competitive result on a hybrid dataset.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hybrid Vision Transformer and CNN for Deepfake Image Detection

  • Mayank Agrawal,
  • Swastik Kaushal,
  • Manoj Diwakar,
  • Neeraj Kumar Pandey,
  • Aditya Joshi,
  • Sanjay Roka,
  • Prabhishek Singh

摘要

The rapid advancement of deep learning models that can synthesize and generate hyper-realistic videos, known as Deepfakes, and their public accessibility have raised concerns about potential malicious uses. Deep learning techniques can now generate new faces, swap faces between two people in a photo or a video, alter facial expressions and facial features. These powerful video and photo manipulation methods can find potential use in many fields. However, they also pose a growing threat to everyone if individuals use them for harmful purposes such as identity theft, phishing, and scams. In this work, we propose a Vision Transformer and CNN hybrid for detecting fake faces. The hybrid model consists of two components: a Convolutional Neural Network (CNN) and a Vision Transformer (ViT). The CNN extracts local features, while the ViT captures and analyzes the global features that CNN might have missed. We trained our model on a combination of 140k Real and Fake Faces and Real and Fake Face Detection datasets and achieved 92.33% accuracy. Our contribution is that we added CNN for local feature capturing to the ViT architecture and got a competitive result on a hybrid dataset.