In the context of brain lesion classification, classifiers can be used to determine the kind of lesion or tumor in the MRI scan. This information can then be used by doctors to make more informed decisions about the patient’s treatment. Classifiers should also be able to handle the variability in the appearance of brain tumors. A comparative study of two classifiers can help medical professionals expedite the classification process by providing insights into each classifier’s advantages and disadvantages. This information can then be used to choose the most appropriate classifier for a given task. In this paper, we explore a novel strategy, SVM binary classification (presence or absence of abnormal mass), to achieve increased accuracy compared to a medically futile multi-class classification and assess the performance against the softmax classifier. First, we use a CNN to extract features from the MRI images. The SVM classifier is known to be robust to noise and overfitting, which makes it a good choice for medical image classification. We evaluated our proposed method on an open-sourced dataset. The data used for this research contains MRI scans of brain tumors, labeled into different categories such as benign, malignant, and pituitary tumors. Our proposed binary classification model, using SVM as the classifier has achieved an overall classification accuracy of 96%, which is superior to the performance of previous methods on this dataset.