One of the worst diseases in recent years is thought to be brain tumors. If it is not identified in its early stages, it also results in death. Numerous deep learning and machine learning methods have been put out to accurately identify the presence of a brain tumor at any stage. A customized convolutional neural network (CNN) is proposed in this paper which has minimum layers as compared to the standalone networks. It has 11 layers which takes less training time and generates the correct class. The proposed model results are supported by the Explainable AI algorithm which gives practitioners a better visualization regarding the predicted class. The Axioms-based Gradient-weighted Class Activation Mapping (Xgrad Cam) is an extended version of Grad-Cam which dominates the areas that contribute to predicting the corresponding class. The proposed model has given us promising results not only on multi-class publicly available brain tumor datasets but also on binary-class brain tumor datasets. Two multi-class brain tumor datasets and one binary-class brain tumor dataset are used for testing the presented technique. These datasets have an imbalanced dataset and the customized model outperforms in those applications. The results are briefly discussed in the experimental results section.