<p>Video-based human action recognition is the popular research areas in the field of computer vision. The emergence of deep learning model is effective in extracting spatial features and local patterns within the video. However, because of the large number of parameters and resource requirements, these methods frequently face difficulties with computational inefficiency. A novel framework for the identification and categorization of human actions utilizing videos is suggested as a solution to these problems. Important steps in the process include pre-processing, segmentation, feature extraction, and activity detection. Initially, the input video is preprocessed using the Gaussian filter to reduce noise and focus on crucial visual content. Subsequently, an Unpooling layer Assisted SegNet (UASN) model is proposed to divide the preprocessed images into meaningful segments. Subsequently, relevant features such as shape, color, and an improved Pyramid Histogram of Oriented Gradients (PHoG) are extracted from the segmented outcome to enhance detection accuracy. Finally, a hybrid model is proposed that combines an Improved Bi-directional Long Short-Term Memory (Bi-LSTM) and LinkNet for action classification. This integrated approach aims to improve efficiency of action detection through advanced deep learning techniques. Furthermore, the analysis of action recognition in videos was conducted using the UCF101 Videos and HACS datasets. The suggested Improved Bi-LSTM+LinkNet method is also compared to the existing methods. As a consequence, the proposed method performs better than the existing methods, obtaining an accuracy of 0.980 and specificity of 0.983 using the UCF101 Videos dataset and an accuracy of 0.978 and specificity of 0.988 using the HACS dataset.</p>