Accurate detection of deepfake media has become a challenging problem with the advent of sophisticated generative artificial intelligence-based methods. Though such media primarily contain audio and visual information, most state-of-the-art detection methodologies do not exploit all modalities and rely on a single modality. A deep fake video can have tampered audio or video or both. In this work, we propose a multimodal deepfake detection model that leverages audio-visual inconsistencies at the stage of early fusion and late fusion. We extract the visual frames and audio from the video. In the preprocessing, we apply reduced Fast Fourier Transform (FFT) on each video frame to capture the most significant frequency components. We compute the mean of the reduced FFT of all frames to represent the video feature. We transform the audio into a mel spectrogram to capture the frequency composition of the audio over time. In early fusion, visual and audio features are stacked together and pass through patch embedding. It is then passed to the dual-head attention network. In late fusion, visual and audio features from the preprocessing phase separately pass through the patch embeddings and transformer encoders to capture the detailed information of each modality. It is then passed to the cross-modal attention network to capture the interactions of both features. The early and late fusion output is combined and passed through the dense layers to classify real and fake classes. The proposed model achieves the testing accuracy of 97% on the benchmark FakeAVCeleb [12] dataset. It also shows comparable results with the state-of-the-art models on the FakeAVCeleb multimodal dataset.