<p>Human activity recognition (HAR) is concerned with identifying human activities gathered using devices like video cameras. The video datasets are challenging in terms of quality: varying backgrounds, camera viewpoints, and huge volume. Deep learning (DL) paradigms are more suitable for such datasets than traditional machine learning techniques. But sometimes DL HAR models suffer from the misclassification problem when tested in real-time environment or on unseen data. To overcome such issues the DL-based ensembled models have proven beneficial and provide high performance. In the proposed work, a deep ensembled voting algorithm (EnsVoting) framework is introduced where the prediction is based on the output of Conv-Rec base models. In Conv-Rec model, a deep convolutional neural network (CNN) is used to extract deep features from videos that are passed to the bi-directional recurrent neural network (RNN) layer later. Then the base model’s prediction is used by the proposed EnsVoting model which provides more efficient results. To validate the trained models’ performance, two validation sets are made (i) seen video set: from the UCF-101 dataset; (ii) unseen video set: from the Kinetics-700 dataset. The proposed work is well researched, comprehensive, and offers novelties (i) EnsVoting algorithm based HAR framework (ii) implemented fast trained HAR model using ResNetRS-152 with 91.25% accuracy, (iii) scientific validation including seen video set with accuracy 96.25% and unseen video set with accuracy nearly 73% and, (iv) evaluated the overall performance in terms of accuracy, ROC/AUC and performed the statistical test for evaluating the p-value. </p>