HoloSpeak: Real-Time Speech-to-Text for Inclusive Communication
摘要
This work aims to build a real-time Automatic Speech Recognition (ASR) system capable of both English and Hindi, with a primary focus on Hindi, with the help of advanced speech to text models along with MAX7219 LED dot matrix display for live text visualization. The system accomplishes transcription by using models like Whisper, Wav2Vec2 and Retrieval-Augmented Generation (RAG). The core functionality of the project consists of capturing spoken Hindi speech through a microphone, processing the speech using the ASR model, and dynamically showing the text displayed from the ASR model on the scrolling LED matrix. It also increased accessibility specifically in social and public communication dimensions by providing a meaningful text representation for people with hearing impairment about speech in real-time. The project advances on speech recognition-based accessibility solutions and extends its use case to LED-based holographic displays. The study covers a comparative analysis of Wave2Vec2 and whisper ASR model and progresses towards integrating the RAG model into the ASR system to improve contextual understanding, improving transcription accuracy, as well as optimizing the system for embedded deployment resulting in holographic preview of floating text, enhancing comprehension for deaf users.