Efficient evaluation for pavement performance has always been a significant problem for road engineers. However, the traditional methods for collecting pavement performance information may be either expensive or low efficient, depending on the automated degree. To address the above challenges, this paper proposed a low-cost, convenient, and intelligent method for pavement performance evaluation using vibration data and ground speed of vehicle. A lightweight intelligent terminal integrated an inertial measurement unit (IMU), a data transmission unit (DTU) and a global positioning system (GPS) was developed to collect various data induced by vehicle–road interaction. After data preprocessing and feature extraction, an unsupervised combined with supervised machine learning method was proposed to decouple the relationship between vehicle vibration, speed and pavement performance. The results showed that the proposed data-driven method clarified three pavement performance ratings. The validity of the proposed method was verified using the evaluation results from the repeated tests. Finally, the visual inspection for these evaluation results was provided on the map. This paper provides a low-cost insight for lightweight pavement performance evaluation, which has potential implications for monitoring the road quality in the field of road asset management.