This research presents a novel hybrid framework combining Ant Colony Optimization (ACO) and ResNet-50 deep learning architecture for efficient breast cancer detection and classification in mammographic images. The proposed methodology leverages the strengths of both bio-inspired computing and deep learning to enhance detection accuracy and reduce false positives. ResNet-50’s pre-trained architecture is utilized for robust feature extraction from mammogram images, while ACO optimizes the feature selection and classification process. The system operates in multiple phases: initial preprocessing enhances image quality through adaptive histogram equalization and noise reduction; ResNet-50 extracts hierarchical features from the preprocessed images; ACO then optimizes the feature selection process by treating the feature space as a graph where artificial ants traverse to identify the most discriminative feature combinations. This hybrid approach demonstrated superior performance compared to traditional methods, achieving 97% accuracy sensitivity on the DDSM dataset.