In this technological age, Sign Language Recognition (SLR) is very critical to improve the communication flexibility for hearing- and speech-impaired people without depending on human interpreters. Video-based classifications of sign gestures are still challenging due to the existence of spatial-wise and sequential-wise complexities. In recent years, numerous efforts have been proposed to mitigate spatial-wise complexities but fewer on sequential-wise complexities. This paper introduces an efficient segmented-clustered keyframe extraction algorithm for the classification of isolated sign gesture videos. The algorithm integrates sequential segmentation and K-Means clustering to retain repetitive signs and eliminate redundant and information-less frames. The method has successfully reduced the size of the dataset to 13%. The method reduces the complexity of the model and hence boosts the performance. The proposed work explores the INCLUDE dataset. Additionally, we conducted a comparative analysis of advanced CNN-RNN hybrid models to identify the optimal model for classifying these gestures. The hybridization of MobileNetV2 and BiLSTM on the INCLUDE dataset has achieved the performance with an accuracy of 79.77%.