The communication gap between doctors and patients with hearing impairment creates significant obstacles in delivering medical diagnoses to patients from Tanzania. In this paper, we introduce a paradigm of assessing Tanzania Sign Language identification by using spatial-temporal feature extraction to bridge this gap. We build a model combining Support Vector Machines (SVM) for classification, Long Short Term Memory (LSTM) networks for temporal modeling, and Convolutional Neural Networks (CNN) for spatial feature extraction. Our model reached 95% accuracy, 96% precision, 94% recall, and a 95% F1 score by analyzing 2,314 video samples of 32 common medical terms. These results surpass previous state-of-the-art methods, including independent SVM or CNN models, which generally achieve 85–90% accuracy in tasks for recognizing sign language. Our methodology surpasses prior benchmarks by 5% in recognition accuracy because it connects spatial and temporal modeling approaches to develop a comprehensive tool for enhancing healthcare services to deaf and hard-of-hearing Tanzanians.