This research introduces a unique approach to the classification of road surface characteristics, which includes the identification of normal road segments, potholes, and fractures. The Whale Optimization Algorithm (WOA) is combined with Convolutional Neural Networks (CNNs) in order to accomplish this classification. Ten percent of the 6,000 photos in the dataset were set aside for testing, 10% for validation, and 80% were used for training the model. Eighty epochs later, the model had a 0.4715 loss and 92.58% accuracy. The hybrid model's performance was validated through comparisons with models such as VGG16, ResNet50, InceptionV3, Xception, and MobileNetV2. The evaluation that was conducted using a confusion matrix and classification report resulted in a macro average F1-score of 0.93. The ROC–AUC values in this case were 0.93 for cracks, 0.97 for potholes, and 0.94 for normal roads. The study highlights the methodology, experiments, and comparative insights, emphasizing the importance of accurate pavement defect detection for road safety and cost efficiency. While deep learning shows promise in detecting pavement cracks, feature selection remains a challenge. The method proposed in this study combines WOA for optimum feature selection, random forest (RF) for classification, and deep learning for feature extraction. It is evaluated using metrics such as F1-score, AUC, accuracy, recall, and precision.