Mel-Spectrogram Reconstruction from Video Lip Sequences Using an Autoencoder Architecture
摘要
Reconstructing Mel-spectrograms from visual input is a crucial step toward generating intelligible and natural-sounding speech in scenarios where the audio signal is unavailable or compromised. Despite recent advances in Visual Speech Recognition (VSR) for English and other high-resource languages, there is a notable scarcity of datasets and models specifically designed for Spanish. In this work, we introduce a novel Spanish-language audiovisual dataset composed of short video clips of isolated speakers, with aligned transcriptions and their corresponding Mel-spectrogram representations. This work proposes a lightweight convolutional autoencoder model designed to reconstruct segments of Mel-spectrograms directly from sequences of grayscale lip region frames. The architecture processes individual frames through a shared convolutional encoder and combines their latent representations to produce a fixed-size spectrogram patch. Experimental results demonstrate the viability of lightweight, language-specific models for visual-to-audio reconstruction, paving the way for future research in silent speech interfaces and robust speech synthesis for Spanish speakers.