MMSP-Net: Phase-Dependent Multimodal Speech Repair Modeling
摘要
This paper introduces a new method for speech restoration by combining image, audio, and text information to repair missing speech content. In the restoration process, the speech signal is first converted into amplitude spectrograms and phase spectrograms. Then, image restoration techniques and text-guided approaches are used to restore the missing speech content. The image restoration techniques are based on deep learning models that fill in the missing image content to restore the speech signal. The text-guided approaches utilize the textual content of the missing parts to guide the repair of the magnitude and phase spectra, enabling accurate speech restoration. This multimodal approach, which combines image, audio, and text, provides a new perspective and technical means for speech restoration. By repairing the missing speech content, the quality and intelligibility of the restored speech can be improved, offering strong support for the development and application of speech restoration technology. Additionally, this method can serve as a reference for other tasks based on multimodal data, such as text-to-image generation. By combining different sources of information, it is possible to more accurately restore missing content and improve the performance of the task.