Accurate classification of benign and malignant tumors in mammography is critical for early breast cancer detection. Traditional diagnostic methods rely on radiologists, but machine learning (ML) and deep learning (DL) techniques have improved accuracy and consistency. However, hyperparameter tuning, which significantly impacts model performance, remains a challenge. This study presents an optimized diagnostic system integrating MobileNet, a convolutional neural network known for efficient image classification, with the Snake Optimization (SO) algorithm, an advanced hyperparameter optimization technique. The mini-MIAS mammography dataset is preprocessed and augmented to enhance robustness. Experimental results demonstrate that incorporating SO significantly improves the MobileNet model’s classification performance. The optimized model achieves a classification accuracy of 92.79%, surpassing the pre-optimization accuracy of 91.68%. Precision for benign and malignant cases increases from 0.91 to 0.92 and from 0.89 to 0.91, respectively, while recall remains high at 0.93. The F1-score for benign cases improves from 0.91 to 0.92, reflecting enhanced reliability in clinical decision-making. These statistically significant improvements align with the medical objective of increasing diagnostic accuracy. The proposed system represents a step toward fully automated mammogram analysis, assisting clinicians in making more reliable diagnoses for early and effective cancer detection.