Detecting events automatically in videos is gaining popularity in the field of surveillance health care, security. In surveillance, intelligent systems are used to detect suspicious activities or individuals to track people in large crowds or public spaces. In healthcare, it is used for early detection of disease and monitoring elderly people’s tendency to fall. Human action recognition (HAR) is a rapidly growing area that focuses on recognizing and interpreting human actions from video data. However, it is challenging to detect complex events due to intra class and interclass variations. The use of deep learning techniques for HAR has resulted in significant improvements in accuracy and efficiency. The paper presents a short summary of the various works reported using deep learning techniques in action recognition and to investigate the performance of cross entropy loss function using ConvLSTM for HAR analysis with increasing class categories. UCF101 data set is considered for experimental investigation.