Person tracking is a critical area in computer vision, with applications in autonomous vehicles and surveillance systems. Despite advancements, real-world challenges such as occlusion, dynamic lighting, and crowded environments hinder the performance of existing methods. This study introduces a robust and scalable person tracking framework integrating Siamese networks for feature extraction and Conditional Random Fields (CRFs) for probabilistic modeling. The Siamese network provides discriminative embeddings, while CRFs ensure temporal and spatial consistency, enabling precise tracking in complex scenarios. Evaluation on the LaSOT dataset, featuring diverse real-world conditions, demonstrates the proposed method’s superior performance over state-of-the-art algorithms like KCF, CSRT, and MeanShift. The system achieves a 75% success rate and an average Intersection over Union (IoU) of 0.65, highlighting its resilience to occlusions and appearance variations. Although computationally intensive, the framework maintains real-time efficiency, making it practical for real-world deployment. This research addresses key limitations in person tracking, laying the foundation for future improvements in multi-object tracking and embedded applications.