With the acceleration of urbanization and the rise of traffic demand, road defects have become a major challenge to traffic safety. Target detection based on deep learning provides an effective solution for pavement defect detection. However, the complex road background, various detection equipment and various road defects make it difficult to balance the spatial location sensitivity and multi-dimensional feature focusing, which limits the detection performance. To address these issues, this paper proposed CMANet (The abbreviations of the two modules are adopted). Meanwhile, the paper constructs a multi-semantic collaborative feature extraction module (CSSA). It can mitigate semantic differences, guide multi-semantic information, and enhance the model’s feature extraction ability. In addition, a novel multi-scale efficient fusion attention mechanism (MAI) is presented. It captures spatial remote location information, dynamically adjusts channel weights and spatial context, thus improving the accuracy and robustness of road surface defect recognition. Finally, the model’s performance is verified on open road surface defect datasets like RDD2022, SVRDD, and IRDD, compared with the original model, CMANet achieves 2.5%, 1.3% and 1% increase in mAP0.5, and the number of parameters is reduced by 1.5%. The experiment results show model’s good cross-domain recognition ability.