Real-time multimodal sign language recognition systems have been developing very fast in the past years. However, the inclusion of facial expressions for the proper interpretation of sign language is still a less researched topic. This paper is concerned with a real-time multimodal sign language recognition system integrating gesture recognition and facial emotion detection. Gesture recognition utilizes a Convolutional Neural Network–Long Short-Term Memory (CNN–LSTM) model with MobileNetV2 for spatial feature extraction and LSTM for temporal learning. With DeepFace, facial emotion detection is realized, using analysis of facial expressions for adding an emotional context to the signs. Evaluation with the proposed architecture on Small Word-Level American Sign Language (WLASL) gave it a high score in gesture classification at an overall 92%, which further improves the effectiveness of the system in gesture recognition and understanding emotional nuances. Despite this success, there remain challenges like the limitation of the dataset and the deployment in resource-constrained environments. The future work is diversifying the datasets, optimizing computational efficiency, and enhancing robustness under real-world conditions. These results are important for multimodal learning to gesture recognition in advancing accessibility and communication for the deaf and hard-of-hearing communities.