3D multi-object tracking aims to track multiple moving objects over time and maintain their identities across frames, with widespread applications in fields such as autonomous driving. However, current studies fail to handle the anomalous detection values and have undesired latency due to complex similarity metrics. We propose RobTrack, a novel multi-object tracking solution to address these issues. First, RobTrack designs a new residual Kalman filter to reduce the negative impact of detection anomalies within the Tracking-by-Detection (TBD) architecture. Second, we design an angle-penalty-based lightweight similarity metric, utilizing the approximation of rotated detection boxes and angle penalties to enhance inference speed and accuracy. RobTrack achieves state-of-the-art performance on the NuScenes (AMOTA 76.6%), KITTI (HOTA 82.9%), and Waymo (MOTA 75.2%) datasets while significantly improving tracking speed and enabling real-time tracking on resource-constrained edge devices, even in object-dense environments. Furthermore, the proposed modules can be integrated into other TBD works, exhibiting competitive performance.