Automatic speech recognition (ASR) based on deep learning has come a long way, thanks to the growing need for speech-to-text transcription in multimedia tools like video subtitling and accessibility tools. While OpenAI’s Whisper model has demonstrated state-of-the-art transcription performance across multiple languages and accents, its real-world deployment remains challenging due to linguistic variations, background noise, and formatting inconsistencies. This study investigates the application of Whisper for automated video captioning, evaluating its baseline transcription accuracy and exploring its limitations under diverse linguistic and environmental conditions. The research focuses on assessing Whisper’s weaknesses, particularly in accented speech recognition and robustness in noisy environments, and proposes a series of post-processing techniques to enhance its usability. The methodology consists of dataset pre-processing, model evaluation, real-world robustness testing, and the application of phonetic normalization, punctuation restoration, and spelling correction. The study result demonstrates that Whisper achieves a Word Error Rate (WER) of 4.75% after post-processing but struggles with Scottish-accented speech (WER 18.18%) and noisy environments. By introducing post-processing strategies, including phonetic adaptation and transcript enhancement techniques, the study significantly improves transcription accuracy, readability, and usability for real-world applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automated Speech-To-Text Captioning for Videos and Noise Robustness Analysis Using OpenAI Whisper: A Performance and Enhancement Study

  • Thanh Thanh Ngoc Nguyen,
  • Manh-Son Tran,
  • Duc-Tho Mai

摘要

Automatic speech recognition (ASR) based on deep learning has come a long way, thanks to the growing need for speech-to-text transcription in multimedia tools like video subtitling and accessibility tools. While OpenAI’s Whisper model has demonstrated state-of-the-art transcription performance across multiple languages and accents, its real-world deployment remains challenging due to linguistic variations, background noise, and formatting inconsistencies. This study investigates the application of Whisper for automated video captioning, evaluating its baseline transcription accuracy and exploring its limitations under diverse linguistic and environmental conditions. The research focuses on assessing Whisper’s weaknesses, particularly in accented speech recognition and robustness in noisy environments, and proposes a series of post-processing techniques to enhance its usability. The methodology consists of dataset pre-processing, model evaluation, real-world robustness testing, and the application of phonetic normalization, punctuation restoration, and spelling correction. The study result demonstrates that Whisper achieves a Word Error Rate (WER) of 4.75% after post-processing but struggles with Scottish-accented speech (WER 18.18%) and noisy environments. By introducing post-processing strategies, including phonetic adaptation and transcript enhancement techniques, the study significantly improves transcription accuracy, readability, and usability for real-world applications.