Video surveillance is used in several domains to monitor a person’s activities toward finding a malicious activity to secure various points. The presence of malicious activity has been detected through video surveillance according to various approaches. The methods use video frames to extract features like shape, size, texture, object, and sketch. Using the features extracted, the method would estimate similarity measures to find the class of the image object. The activity monitoring can be performed by maintaining different templates, shapes, textures, and sketch features. The method would estimate similarity measures against the samples of various activity classes. This article focused on conducting a detailed survey on monitoring malicious activity through video surveillance frames. By capturing the video frames and extracting the features by segmenting them, they can be maintained under various classes of activities. Further, the same features can be extracted from the test image and used to measure the similarity toward different activity classes. According to the similarity value, the method would select the most dominant activity class as a result. Accordingly, there are several deep learning algorithms available: Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Radial Basis Function Networks (RBFNs), Multilayer Perceptrons (MLPs), Self-Organizing Maps (SOMs), Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs), and Autoencoders. This article presents a detailed analysis of different approaches to the problem.