Pavement distress detection is crucial in urban road maintenance. Traditional detection method such as manual inspection suffers from heavy workloads and cost, and is being replaced by image recognition technologies based on convolutional neural networks (CNNs). It is expected that this technique could cover both images collected by manual and machine inspections. However, the ambient diversity from manual shot images and auto shot images can influence the network training, then leads to accuracy decrease. This paper hence proposes a Bilevel Spatial Attention Module (BSAM) in YOLOv5 network to enhance the adaptability of CNN for different shooting environments. Combining both bi-level routing and spatial attention mechanisms, BSAM can collect the location and feature of the target simultaneously from the image, hence improve the detection performance. The method has been successfully applied in the urban road maintenance system of the local city. The distress images of the city roads captured in last three years are used as datasets. Results show that the YOLOv5-BSAM method could accurately localize the pot hole, map crack and strip crack on pavements. Comparison study demonstrates that comparing with other attention mechanisms, the proposed method could improve the accuracy by 5%~10%.