One crucial first step toward early management is the accurate categorization of brain tumors. Recent years have seen the adoption of artificial intelligence (AI)-based diagnostic technology to automate the process and provide faster, more accurate diagnosis. In order to improve brain tumor categorization, the paper presents an upgraded AI-based architecture. In order to produce an ensemble classifier and a more reliable framework for classifying brain tumors, we introduce a hybrid architecture that combines vision transformer (ViT) with deep neural networks. Preprocessing and data normalization are the first steps in the analysis pipeline. Three feature sets are then taken out for the categorization process. Using a histogram of oriented gradients (HOG), the first module serves as a descriptor for the morphological (structure/shape) attributes. Additionally, local deeper information is extracted from the brain images using convolutional neural networks(CNNs) with optimized design. Lastly, because ViT-derived features are less susceptible to data argumentation and can manage dependencies over longer distances, they are also included. To classify brain MRIs, the retrieved features are fused and a machine learning classifier is employed. Our ensemble system has been tested on four-classes publicly-available MRIs dataset of the brain (7,023 total images). The results, using various metrics, showed that leveraging the benefits of individual modules of our design lead to improved performance using ablation studies.