Robust estimation of on-road vehicle positions is important for traffic monitoring. We propose a neural network based approach to recover the ground locations of vehicles in traffic surveillance videos. We frame the location estimation problem as an object detection task. In contrast to the current state-of-the-art image object detection methods which only roughly estimate the object bounding boxes on the image plane, we aim to estimate the vehicle locations on the ground (the location in the frame of the road ground plane). In order to achieve this goal, we first transform the image into the bird’s eye view (BEV) image, and then detect the bounding box of vehicles in the BEV images. However, the BEV images exhibit severe distortions which hinders the detection accuracy significantly. Furthermore, the detections from consecutive images are inconsistent, i.e. the bounding boxes of the same vehicle in different images are of different sizes. To address these two challenges, we present a multi-task detection model which significantly boost the detection rate, and we design a novel consistency loss which constraints the size of detection boxes of identical vehicles to be the same. In order to effectively train and evaluate the model, we collect a dataset of traffic videos, which contains synchronous videos captured from both frontal view and bird’s eye view. Our experiments show that our multi-task detection model and consistency loss significantly improve the detection accuracy and stabilize detection results.