This study examines the effectiveness of Convolutional Neural Networks architectures such as VGG-16 and the customized convolutional neural network in classifying brain tumors using medical imaging data. It explores the impact of various convolutional layers and hyperparameters on classification performance, considering factors such as training accuracy, validation accuracy, confusion matrix, and F1 score. The VGG-16 model, despite having more parameters than other models, achieves a 16% higher F1 score in medical image classification tasks. The insights from this study can aid researchers and practitioners in the field of medical image analysis in automating tumor classification. The insights may be useful for researchers in determining the base model for brain tumor classification that has a trade-off between the accuracy and computation time. Also, the fine-tuned model can be integrated with a mobile or web application for automating the tumor classification.