Mobile AI Application for Continuous Sign Language Recognition
摘要
In an era where significant developments have occurred in spoken language translation with systems available at the click of a button, sign languages translation remain largely excluded. This work proposes a mobile application that enables the translation of sign language into verbal language in real-time and as accessible as translations between verbal languages. In this communication process, the mobile device works as a channel, where the sender (signer) is recorded while communicating his message in sign language, and the receiver (non-signer) obtains the translated message through a display or sound device. The proposed system uses Google MediaPipe framework to extract landmarks from video frames, an adapted Transformer model to translate these landmarks into glosses, and a fine-tuned GPT4o-mini model to convert the glosses into verbal language, in this case German or English. Two Transformer models were trained using the German Sign Language Database SIGNUM. The first Transformer model enables the translation of pre-recorded videos stored on the mobile device. The second model, was trained with a custom extended version of the SIGNUM Database through the isolated signs. This second model supports real-time translation of messages captured by the mobile device. This real-time system classifies the message every second while maintaining memory of prior inputs to ensure coherent translations. These classifiers make it possible to develop a human-computer interaction application for a mobile device to facilitate and enhance communication between individuals.