This study presents a computationally efficient Convolutional Neural Network (CNN) enhanced with transfer learning for medical image classification. The method was tested on three tumor datasets: brain MRI, lung and kidney CT scans. It leverages a pre-trained CNN on brain MRI images, fine-tuned with minimal re-training for the CT scans, achieving high classification accuracy. Transfer learning allows the model to adapt to cancer-specific features by utilizing insights from large datasets. Re-training on each tumor type using only 20 epochs can deliver significant classification performance, demonstrating the method's efficiency. The CNN's computational efficiency ensures it is both accurate and scalable, making it suitable for use in resource-constrained environments. This research highlights the potential of low-complexity deep learning (DL) models to accelerate cancer diagnosis while balancing accuracy and efficiency. It shows that complex deep learning models are not always necessary, and optimal performance can be achieved with lower computational costs.