One of the major healthcare challenges is the classification of brain tumors, howbeit, early and exact detection of tumor classes such as glioma, meningioma, and pituitary tumors can immensely enhance the patient’s prognoses and be of great help in the therapy planning. The catch is that conventional deep learning models, while they are efficient, tend to weigh on the excessively complicated architectures and have limited interpretability, thus making them unsuitable for clinical scenarios that require transparency, trustworthiness, and computational efficiency. In addition, these models might wrongly support that the changes they find in the tissue (which actually can be the normal soft tissue or even other underlying or neighboring structures) are the cause of the diagnoses that they arrived at, thus, leading to the loss of the diagnostic trust and possibly misclassification. This investigation has come up with a pioneering conceptual framework called proposed framework which links the Explainable Artificial Intelligence (XAI) methods to the simplified layout of the Convolutional Neural Network (CNN). The primary objective of this structure is to improve the diagnostic accuracy and the model interpretability as well by figuring out an interaction-emphasizing portion(s) of the brain MRI scan that, in turn, leads to the identification/diagnosis decision made. In the first place, proposed framework uses Gradient-weighted Class Activation Mapping (Grad-CAM), SHapley Additive exPlanations and Local Interpretable Model-agnostic Explanations (LIME) to present the feature importance in a pictorial manner and direct the architectural refinement. Later, they utilize these revelations for the elimination of those layers from the CNN, which not only contribute very little to the final classification but also great computational overhead is generated from them. Hence, the computational power will be lessened to the necessary minimum while the performance of classification will remain almost unchanged. This model is trained and tested on two open-sourced brain MRI datasets, implementing the thorough preprocessing steps like converting to grayscale, cropping morphologically, normalizing, resizing, and data augmentation. The training process is done in an optimal way with the help of Adam optimizer and sparse categorical cross-entropy loss, and overfitting prevention is through early stopping. The performance measures comprise accuracy, precision, recall, and F1-score, which are calculated for both the training and the test sets. The research findings to date validate the claims of this idea, proposed framework scoring 99.21% accuracy on the primary dataset and 94.72% on the secondary dataset that has never been seen before, thus, it is propelled ahead of the latest powerful models in terms of efficiency as well as interpretability. Furthermore, the evidence produced by Grad-CAM and SHAP serves to assure that the tumor areas are indeed the focus of the model’s attention, and thus appropriate for use in the clinic. These victories point to the framework’s strength, and its ability to work with various kinds of data, thus, making it a perfect fit for deployment in real-life situations particularly in the less developed areas. Coupled with transparent decision-making, the high precision of proposed framework sets a new milestone in medical image analysis and augurs the future of interpretable AI-driven diagnostics in this space.