As the demand for road inspection continues to increase, traditional manual inspection methods are insufficient for large-scale road inspections. Researchers continue to propose new detection methods for road anomalies. However, existing detection methods are not effective enough because of the varied types, scales, and complex backgrounds of road anomalies. Therefore, this paper proposes a road anomaly detection method RADNet-FG based on deep learning, aiming to assist inspection personnel in completing routine inspections and key inspections.First, we propose a RADNet algorithm for routine inspections, including Deformable Feature Extraction module (DFE), Coordinate Attention module (CA), and Directly Connected Feature Pyramid Network (DCFPN). Then, we cascade the Fine-grained Classification module (FG) on the RADNet algorithm to form RADNet-FG algorithm, which enhanced the classification ability of RADNet. Finally, experiments on our datasets show that in routine inspections, RADNet algorithm achieves a precision rate of 93.1%, a recall rate of 86.5%, and a mAP of 91.6%, which is better than Faster R-CNN, YOLOv8, YOLOv9, etc.; In key inspections, RADNet-FG algorithm achieves a precision rate of 95.6%, a recall rate of 48.3%, and a mAP of 90.1%, which are 5.0%, 3.5%, and 15.4% higher than RADNet without fine-grained classification, and are 10.6%, 2.2% and 21.3% higher than YOLOv8.