Vision-Based Sign Language Recognition and Multilingual Translation for Facilitating Deaf and Mute Communication
摘要
Sign language serves as the primary means of communication for individuals who are both deaf and mute. Nevertheless, communicating with those who do not understand sign language poses a significant challenge. Due to the structural differences between sign language and written/spoken languages, a communication barrier exists. Consequently, the interaction between deaf and mute individuals heavily relies on visual-based communication. To address this issue, a vision-based interface system has been developed to facilitate communication between deaf and mute individuals and the broader public. This system offers an interface capable of translating sign language gestures into text, enabling those unfamiliar with sign language to readily comprehend the message. The proposed system involves real-time video analysis for sign language identification and recognition, followed by the conversion of these visual inputs into English and other native languages. In this paper, French and Japanese languages are supported through Google API translator services, utilizing the SSD MobileNetV2 model for sign language recognition. The system achieved an outstanding overall accuracy of 0.98, with individual sign tokens demonstrating average accuracy scores of 0.99 for “Hello,” 0.99 for “I love you,” 0.98 for “Thank you,” 0.97 for “Yes,” 0.96 for “No,” and 0.96 for “Help.”