Finger Text: Deep Learning for Fingerspelling Translation
摘要
This research proposes an innovative solution for detecting and translating Sign Language fingerspelling into text, addressing accessibility challenges for the Deaf and hard-of-hearing community. The model combines an enhanced Squeeze-former for feature extraction and a Transformer Decoder for sequence decoding, outperforming traditional ASR and machine translation technologies. Augmentations such as Cut Mix, Finger Dropout, and Time Stretch enhance generalization, while a diverse dataset of three million fingerspelled characters in real-world scenarios ensures superior performance. The 3 channel image interpretation, Llama attention, and optimized training allow for efficient deployment of deeper models. Transformer-based decoding is emphasized over CTC-based approaches, with a confidence score aiding in identifying corrupted examples during postprocessing. Training insights, including a cosine learning rate schedule, mixed precision, and tf-lite inference, demonstrate robustness and scalability. Post-processing techniques, incorporating confidence scores, improve accuracy, and the paper concludes with an ensemble approach, highlighting the minimal impact of supplemental data. Overall, the research significantly advances sign language recognition, promoting inclusivity in line with the mission of universal AI accessibility.