<p>With the rapid development of artificial intelligence (AI) techniques, computer vision (CV) has become a crucial technology that can replace human supervision in many applications, such as intelligent traffic systems (ITS), healthcare monitoring, security, and military. The timely detection of traffic anomalies through surveillance videos can be highly effective in managing traffic safety in cities and urban areas. Traffic anomalies are broadly categorized into three classes, such as pre-event, current-event, and post-event. Several CV models are developed for detecting various traffic anomalies. However, there are no methodological surveys to provide detailed insights into the pre-event, current-event, and post-event contexts. To understand the main development status of ITS, we have analyzed existing CV models to detect all these traffic anomalies and corresponding benchmark datasets. The creation of CV models based on the concept of granulation is an example of this. This study includes both pedestrians’ behavior-based and drivers’ behavior-based pre-events. It also reveals vital gaps in the available datasets and anomaly detection capability in various contexts, giving future directions to AI-guided traffic anomaly detection research. Challenging issues, including the relevance of granular computing in the ITS domain, are elaborated as a future scope of research, together with some concerns. As anomaly detection is an important step in automatic road traffic surveillance, this survey can be a useful resource for researchers interested in solving various issues of ITS. An extensive bibliography is provided to help researchers with the most influential, recent, or foundational research in the ITS domain.</p>