<p>The serious form of obstruction urban cities face, which is a cause of traffic congestion, creates a need to monitor and regulate real-time situations. In this research, a novel framework is proposed that applies deep learning-based object recognition techniques along with machine learning techniques to provide accurate prediction at the level of congestion. The system is designed to monitor video frames in real time through various processes including camera calibration, region of interest detection, and frame enhancement. The Faster R-CNN algorithm is used to detect the vehicle, and movement tracking of a vehicle along with detecting patterns of movement is done with the Kalman filter tracker. The data acquired on vehicles will be used for training a machine learning model where the Random Forest algorithm achieves 99% accuracy in congestion prediction. The experiments' results indicate that the Average Precision by Convolutional Neural Networks exceeds Fast R-CNN by 16.7Among the classification models, Random Forest outperforms SVM with an accuracy of 96%, KNN with 89%, and LR with 78%, showing a 3% margin above SVM and 21% margin over LR. In comparison to the state-of-the-art models like YOLOv6 and YOLOv7, which score 92–94% on an average for the classification of traffic congestion, the approach presented here scores 2–4% better and hence is more reliable in real-world scenarios. Moreover, the Faster R-CNN is relatively balanced between the speed and accuracy of detection as compared to models based on YOLO, hence, it should become a promising candidate for real-time applications. Findings in the work have demonstrated that a potential way of reaching efficient urban traffic management lies in combining deep learning and machine learning approaches. Such efficiency can allow data-driven measures targeted at reducing congestion.</p>