People who are deaf or who want to remain silent often find it challenging to engage in conversation with others. They may experience a decline in self-esteem and a sense of isolation as a result. By converting sign language movements into text or speech, sign language systems strive to help people with hearing loss communicate more effectively. Presently, methods for identifying sign language rely on a combination of motion translation into text or speech and machine learning algorithms. Many problems exist with the current system, such as the lack of training data and the complexity and unpredictability of sign language. We offer a system that can recognise sign language more accurately and make it more accessible. For precise sign language recognition, it uses cutting-edge deep learning techniques. The system's primary goal is to construct three distinct models—EfficientNetV2, EfficientNetV2L, and the ConvNeXtLarge algorithm—by decomposing video input into its component picture frames. A dataset consisting of 18 dynamic hand gestures executed by 18 participants in an uncontrolled environment is used to assess the proposed system. To train models to recognise and understand sign language movements, it is necessary to feed them the dataset. In a comparison of three models, the ConvNeXtLarge model outperforms the others with an accuracy of approximately 94.67%. When the model is finished running, the next step is to have a translator convert the result to the local language. Traditional Explainable AI models are beaten by the suggested Deep Sign Speak model.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Deep Learning for Real-Time Gesture Recognition and Cross-Lingual Translation

  • Abhishek Kumar,
  • Rishabh Deol,
  • Abhinav Raj,
  • Anuj Kumar Singh

摘要

People who are deaf or who want to remain silent often find it challenging to engage in conversation with others. They may experience a decline in self-esteem and a sense of isolation as a result. By converting sign language movements into text or speech, sign language systems strive to help people with hearing loss communicate more effectively. Presently, methods for identifying sign language rely on a combination of motion translation into text or speech and machine learning algorithms. Many problems exist with the current system, such as the lack of training data and the complexity and unpredictability of sign language. We offer a system that can recognise sign language more accurately and make it more accessible. For precise sign language recognition, it uses cutting-edge deep learning techniques. The system's primary goal is to construct three distinct models—EfficientNetV2, EfficientNetV2L, and the ConvNeXtLarge algorithm—by decomposing video input into its component picture frames. A dataset consisting of 18 dynamic hand gestures executed by 18 participants in an uncontrolled environment is used to assess the proposed system. To train models to recognise and understand sign language movements, it is necessary to feed them the dataset. In a comparison of three models, the ConvNeXtLarge model outperforms the others with an accuracy of approximately 94.67%. When the model is finished running, the next step is to have a translator convert the result to the local language. Traditional Explainable AI models are beaten by the suggested Deep Sign Speak model.