We propose a novel method that detects sources of high vertical acceleration on the road by visual tracking of a preceding vehicle. The method is general, predicting any kind of road anomalies, such as potholes, bumps, debris, etc., unlike direct observation methods that rely on training visual detectors of those cases. The method works in poor visibility or when the preceding vehicle occludes the road anomaly. The approach is validated on our dataset that includes anomalies collected in both controlled settings and real-world scenarios captured in normal traffic conditions. The experiment confirms a strong correlation between a signal measured by a visual tracker, which estimates a vertical displacement of the preceding vehicle, and the IMU signal from the ego vehicle. We demonstrate that our system detects road surface anomalies with high accuracy, achieving AUC 0.969. The method is computationally cheap and runs in real-time on consumer hardware.