<p>Managing incidents along highways is essential in the maintenance of the negative impact on traffic movement and safety, hence the need for efficiency in detection systems as a way of accident control. Existing systems for the investigation of road events are unable to cope with the requirements of the actual state of the art—near real-time operation—while maintaining accuracy in the determination and assessment of risks encountered. This paper describes the problem and its solution in detail by presenting a complete system for advanced highway incident detection and risk classification based on computer vision and deep learning technologies. This technique comprises key frame extraction for reduction of data processing, utilization of state-of-the-art object detection, YOLOv8, and real-time multiline object tracking even in the presence of heavy traffic using the Deep SORT multi-object tracker. Furthermore, several features, including Time-To-Collision (TTC), Post-Encroachment Time (PET), Deceleration Rate to Avoid Collision (DRAC), and headway distance, are extracted. These parameters utilize characteristics such as speed, acceleration, and lane position of the tracked objects, providing an objective risk assessment. Separately, temporal patterns of the incident datasets are enhanced through the incorporation of a Temporal Convolutional Network Auto Encoder (TCNAE) with dilated convolutions and a Bi-directional Long Short-Term Memory (Bi-LSTM) network for effective anomaly detection, while risk levels are assessed through a Deep Dual-channel Convolutional Neural Network (DDCNN) that is spatially aware by classifying the risks in high, medium, and low. The methodology proposed is tested on the highway incident detection data set with good accuracy, sensitivity, and specificity results, which support its applicability that involve monitoring highways.</p>