People with speech impairments or hearing loss frequently utilize sign language as a way of communication. Yet, the language is not understood by everyone. Communication between the deaf and the general public will be facilitated by the automatic translation of sign language into the alphabet or text. This paper proposes a mobile application that enables two-way sign language translation for seamless communication. We leverage deep learning, specifically YOLO v8 and YOLO v5 models, for accurate sign language recognition and translation. Our results indicate that YOLO v8 achieves high precision, recall, and mean average precision (MAP) scores of 98%, 97%, and 99%, respectively, on the Sign Hands dataset, and 95%, 96%, and 98%, respectively, on the American Sign Language dataset. Similarly, YOLO v5 demonstrates competitive performance with precision, recall, and MAP scores of 96.5%, 96.7%, and 98%, respectively, on the Sign Hands dataset, and 94%, 95%, and 98%, respectively, on the American Sign Language dataset. The system offers two key functionalities: (1) Text/Speech to ASL, which converts spoken or written English to text and then translates it into ASL animations using a 3D avatar, and (2) ASL to Text/Speech, which recognizes ASL gestures captured through the camera and translates them into text or spoken English. This mobile application has the potential to improve accessibility for both deaf and hearing communities.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Two-Way Sign Language Interpretation

  • Mahmoud Y. Shams,
  • Alaa Hussien,
  • Hagar Ashraf,
  • Nada Nasser,
  • Mariam Gabr,
  • Hosam Mohamed,
  • Atef Yasser,
  • Roheet Bhatnagar,
  • Tarek Abd El-Hafeez

摘要

People with speech impairments or hearing loss frequently utilize sign language as a way of communication. Yet, the language is not understood by everyone. Communication between the deaf and the general public will be facilitated by the automatic translation of sign language into the alphabet or text. This paper proposes a mobile application that enables two-way sign language translation for seamless communication. We leverage deep learning, specifically YOLO v8 and YOLO v5 models, for accurate sign language recognition and translation. Our results indicate that YOLO v8 achieves high precision, recall, and mean average precision (MAP) scores of 98%, 97%, and 99%, respectively, on the Sign Hands dataset, and 95%, 96%, and 98%, respectively, on the American Sign Language dataset. Similarly, YOLO v5 demonstrates competitive performance with precision, recall, and MAP scores of 96.5%, 96.7%, and 98%, respectively, on the Sign Hands dataset, and 94%, 95%, and 98%, respectively, on the American Sign Language dataset. The system offers two key functionalities: (1) Text/Speech to ASL, which converts spoken or written English to text and then translates it into ASL animations using a 3D avatar, and (2) ASL to Text/Speech, which recognizes ASL gestures captured through the camera and translates them into text or spoken English. This mobile application has the potential to improve accessibility for both deaf and hearing communities.