Communication, a fundamental need, is taken for granted in today’s speaking-and-listening-oriented world. However, for the deaf-mute community that relies on sign language, effective communication with the broader society remains a challenge. One potential solution to fill this gap is the use of computer vision for sign language recognition. Previous efforts to create datasets for Pakistani Sign Language (PSL) have been limited to one frame per sign, missing the dynamic body movements that are crucial for accurately interpreting sign language. Moreover, existing datasets focused solely on alphabets and basic digits, making them impractical for recognizing complete words. To address these challenges, our team developed a novel word-level PSL video dataset, representing a significant step forward in this domain. The dataset ( https://www.kaggle.com/datasets/jahanzebnaeem/wlpsl ) is the most extensive publicly available video dataset for PSL, featuring 31 word classes and 248 videos. Body movement patterns were extracted from the videos to train multiple models for PSL recognition. Recurrent Neural Network (RNN) architecture was selected for its ability to capture sequential dependencies, and specialized versions—Bidirectional-RNN (BRNN), Gated Recurrent Unit (GRU), and Long Short-Term Memory (LSTM)—were employed. The results show the significant potential of our approach. With 15 classes, the LSTM model achieved the highest performance, delivering an impressive validation accuracy of 97% and a validation loss of 0.2. These findings underscore the value of using video datasets to capture dynamic body movements, paving the way for more robust PSL recognition systems.