Purpose <p>Early identification of Cervical Cancer helps in reducing the death rate of CC patients. Another popular technique for identifying cervical cancer early on is a Pap smear/Pap test. Detecting and classifying Pap smear images automatically is a challenging issue. Thus, it’s critical to establish and execute a low-cost, high-efficiency screening system.</p> Methods <p>As a result, we proposed a deep learning system to classify Pap smear images into many classes. In this research, we propose an integrated deep-learning model for the classification of cervical cancer. We used noise-reduction and contrast-enhancement techniques to improve image quality during data pre-processing. The local and global features of the enhanced image are extracted using the modified AlexNet model. The whale optimization algorithm (WOA) is used for feature selection to reduce feature dimensionality.</p> Results <p>Finally, multi-class classification of Pap smear images is achieved using the hybrid ResNeXt-GRU (HResNext-GRU) model, and the Golden Jackal Optimization Algorithm (GJOA) is used to optimize the classification network’s hyperparameters. Herlev and SIPaKMeD, two distinct Pap smear datasets, are used to train and assess the proposed approach.</p> Conclusion <p>Our hybrid deep learning-based system outperforms other recent deep learning models in multi-class classification of cervical cancer, achieving accuracies of 99.80% on the SIPaKMeD dataset and 99.27% on the Herlev dataset. This indicates that the proposed system is robust in classifying cancerous cells.</p>