This paper introduces a new method for speech restoration by combining image, audio, and text information to repair missing speech content. In the restoration process, the speech signal is first converted into amplitude spectrograms and phase spectrograms. Then, image restoration techniques and text-guided approaches are used to restore the missing speech content. The image restoration techniques are based on deep learning models that fill in the missing image content to restore the speech signal. The text-guided approaches utilize the textual content of the missing parts to guide the repair of the magnitude and phase spectra, enabling accurate speech restoration. This multimodal approach, which combines image, audio, and text, provides a new perspective and technical means for speech restoration. By repairing the missing speech content, the quality and intelligibility of the restored speech can be improved, offering strong support for the development and application of speech restoration technology. Additionally, this method can serve as a reference for other tasks based on multimodal data, such as text-to-image generation. By combining different sources of information, it is possible to more accurately restore missing content and improve the performance of the task.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MMSP-Net: Phase-Dependent Multimodal Speech Repair Modeling

  • Xin Ruan,
  • Wenguang Zheng,
  • Wenke Xv,
  • Jin Yang

摘要

This paper introduces a new method for speech restoration by combining image, audio, and text information to repair missing speech content. In the restoration process, the speech signal is first converted into amplitude spectrograms and phase spectrograms. Then, image restoration techniques and text-guided approaches are used to restore the missing speech content. The image restoration techniques are based on deep learning models that fill in the missing image content to restore the speech signal. The text-guided approaches utilize the textual content of the missing parts to guide the repair of the magnitude and phase spectra, enabling accurate speech restoration. This multimodal approach, which combines image, audio, and text, provides a new perspective and technical means for speech restoration. By repairing the missing speech content, the quality and intelligibility of the restored speech can be improved, offering strong support for the development and application of speech restoration technology. Additionally, this method can serve as a reference for other tasks based on multimodal data, such as text-to-image generation. By combining different sources of information, it is possible to more accurately restore missing content and improve the performance of the task.