Video traffic surveillance has become an essential tool for various applications, including security, transportation planning, and traffic management. Recent advancements in deep learning have opened new possibilities for enhancing the performance of vehicle detection and tracking in these systems. This paper addresses the challenges of online action detection in surveillance scenarios by focusing on enhancing multi-object tracking (MOT) performance. Recognizing the limitations of current MOT methods in handling real-world surveillance complexities, we propose a methodology that integrates appearance model extraction directly from the object detector, adaptive adjustments of confidence thresholds and input resolutions, and the incorporation of color information into ReID embeddings. We aim to bridge the gap between motion-based and ReID-based tracking methods, improving both speed and accuracy. Our proposed techniques, including scene-based and object-based adaptation through reinforcement learning, and advanced feature fusion for ReID, are designed to enhance robustness and efficiency. We evaluate our methodology using publicly available datasets, focusing on surveillance-specific challenges. The enhancement in MOT performance is challenging and paving the way for more reliable and efficient surveillance system.