Background <p>A significant challenge in medical imaging is the detection of breast cancer when using mammography images. The early detection of the tumor is an essential step in increasing the patient’s probability; image-processing techniques through advanced algorithms have been very instrumental for this purpose. Purpose: The following research elaborates a novel approach for developing a 3D detection system and GUI visualization of breast cancer by incorporating enhanced algorithms, namely Iterative Closest Point (ICP) and Elastic Convolved ICP (ECICP) for surface registration and the Linde–Buzo–Gray (LBG) algorithm for image segmentation, hence improving the accuracy and efficiency in breast cancer classification using mammograms and MRI images. Material and Methods: The proposed system includes mid-sagittal plane detection, noise removal by improved median filtering, global surface matching, and fine adjustments of mammograms and MRI images. Segmentation and classification are done with the help of an LBG algorithm and knowledge-based techniques for the segmentation and classification of images into normal or cancer-affected ones. The model was trained and tested on a private dataset comprising 1,020 mammograms from 255 patients (133 malignant, 117 benign, and 5 normal cases). Results: Experimental results demonstrated the efficacy of the developed methodology on this challenging data set, with a high value of classification accuracy equal to 97.4%, showing the efficiency of the proposed method for breast cancer detection using the combined use of these three algorithms of ICP, ECICP, and LBG. Conclusions: The conclusions drawn from the research show that advanced algorithms integrated for the precise and efficient detection of breast cancer can be very helpful.</p>