Breast cancer is one of the most common types of cancer worldwide. Mammography has been shown to be effective in detecting and categorizing cancer cells in breast tissues. Several image processing techniques and deep learning models use mammography to develop models with exceptional accuracy. The goal of this paper is to construct deep learning-based models for classifying breast tumors. The mammography image is first pre-processed by using the constrained limited adaptive histogram equalization (CLAHE) and the polynomial curve fitting approach. The second step is to process it through deep learning-based models. The results show that the approach yields a noticeable accuracy for MIAS, INBreast, DDSM, RSNA, and other combinations of data sets.