Most interactions in the contemporary life happen through social networks and social media. We live in a digital world where information is passed in very less time. Most of the information is present in the form of images and videos. But, the authenticity of such data is at stake. Especially, with the threat of DeepFakes looming over digital media, there is a need to develop effective system to encounter it. The current detection systems tend to be biased towards the particular datasets and often fail when the real world data is exposed. In this paper, we propose a novel way to detect facial forgery which brings explainablity and also efficacy. Gabor filters are found to be discriminative by choosing the right set of parameters. Hence, learnable Gabor filters are combined with the EfficientNet model along with channel attention. And also wavelet based efficient net has also been designed. The proposed learnable Gabor filter based Efficient Net has outperformed the existing methods on the Wild DeepFakes dataset and the standard FaceForensics++ dataset. Various cross dataset evaluations have been performed and the proposed methods are found to be effective in generalization.