In today’s crime-stricken world, it has become a necessity for security and surveillance systems. The current surveillance systems are flawed as they require human operators who are susceptible to making errors. Even the smallest oversight can be extremely dangerous. A solution for this issue would be a smart surveillance system that can detect threats. Current solutions are limited due to the lack of large diverse datasets for training and due to the lack of generalizability of the solutions. This paper proposes a method that uses a ConvLSTM model along with incremental learning. The ConvLSTM model ensures that the spatial and temporal information is captured from the data. The use of incremental learning demonstrates how the issue of insufficiently sized datasets for threat surveillance can be overcome by continuously updating the model with new data points by passing new data into the model.