Over the past few years, substantial improvements in AI, ML, and DL have led to the creation of powerful tools for manipulating multimedia. While this knowledge has legitimate uses in entertainment and education, it is also being exploited for malicious purposes. This misuse has given rise to “DeepFakes,” highly realistic and deceptive fake multimedia content, including images, videos, and audio. DeepFakes are used to spread misinformation, incite political discord, and carry out activities like harassment and blackmail. The realistic nature of DeepFakes makes it difficult for humans to distinguish them from genuine content. We propose a technique that takes images or videos as input to form successive target frames, which is fed to the CNN-Swish-RNN, an optimized convolutional RNN network for training and classification. This proposed method helps to identify the real and fake image or video from the input. To assess the robustness of our proposed model, we used datasets from two different sources from Kaggle. We achieved 98.2% accuracy on the selected image dataset. In contrast, we attained 93.6% accuracy using the aggregated video datasets. The proposed technique offers more significant results than existing techniques.