This paper introduces an innovative fusion strategy for 3D object detection, combining detection and segmentation inference for reduced false positive rates. The methodology involves a two-step approach: first we employ a geometric filter to the 3D segmentation outputs confined by the detected bounding boxes, and then utilize an ensemble technique to fuse and refine the detections. By integrating detailed local point-level characteristics from segmentation with the detector’s higher-level bounding box labels, our technique offers a lowered false positive rate. Several existing state-of-the-art methods encounter challenges due to the disparity between training on standardized datasets and real-world deployment. This approach often leads to results in degradation through increased false negatives, which impacts the tracker performance. To address these issues, we propose a novel spatiotemporal filtering algorithm called the Add-Drop Re-Identification Tracker (ADRIT), which effectively rectifies tracking ID switches and detection drops without the need for consistent GPS/IMU information, thereby enhancing performance for tracking-by-detection approaches. In the validation set from the KITTI and NuScenes datasets, our proposed technique exhibits a reduction in false positives while achieving a modest 2–3% accuracy enhancement compared to the baseline model. Our refinement methods are robust against sensor errors where SOTA trackers fail to sustain their performance. Our code is available at github.com/sandeshrjain/3d-det-trk-refine/tree/main