<p>To address challenges such as variations in lighting, weather, and the size and shape of cracks and potholes, we propose an enhanced end-to-end regression algorithm for autonomous road damage detection. This method balances computational efficiency and accuracy by incorporating feature extraction structures to improve performance in scenarios involving multiple damage types, shadows, and fine-grained feature variations. The proposed model integrates a down-sampling structure for dimensionality reduction and feature extraction, an inverted residual mobile block for feature fusion, and an attention mechanism with multi-scale features for multi-scale detail extraction. Additionally, the integration of a Decoupled Head structure enhances bounding box localization. Experimental results show that the proposed method outperforms YOLOv5s (You Only Look Once version 5 small), achieving a 2.9% improvement in the F1 score and a 4% improvement in the mean average precision. Further validation through visualization experiments in seven challenging road scenarios, including varying lighting and environmental conditions, highlights the model’s superior detection accuracy, completeness, and robustness.</p>