Cross-lingual speech synthesis is an active research area in speech synthesis. However, the intricate interplay of linguistic, emotional, and timbral features in cross-lingual scenarios poses significant challenges, often resulting in non-native accents and limited emotional expressiveness in generated speech. These issues are particularly pronounced in low-resource languages due to the scarcity of high-quality datasets and specialized modeling techniques. To address these challenges, we propose ZSEmo-MTVITS, a zero-shot cross-lingual emotional speech synthesis model for Mandarin and Tibetan based on the VITS. We constructed a novel Tibetan emotional speech dataset containing 2500 samples of the Lhasa dialect, covering five emotions (angry, happy, neutral, sad, and surprised), and evaluated Tibetan emotional dataset using the established end-to-end model VITS. Furthermore, we optimize the original VITS model by introducing extra emotion, language, speaker labels and a pre-trained timbre converter model to enable zero-shot cross-lingual emotional speech synthesis for Mandarin and Tibetan. The experimental results show that the ZSEmo-MTVITS model outperforms the baseline model in subjective evaluations, demonstrating superior emotional speech synthesis performance. The model effectively achieves Zero-Shot cross-lingual emotional speech synthesis for Mandarin and Tibetan.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ZSEmo-MTVITS: A Zero-Shot Cross-Lingual Emotional Speech Synthesis Model for Mandarin and Tibetan Based on VITS

  • Weizhao Zhang,
  • Mengjuan Wang,
  • Hongwu Yang

摘要

Cross-lingual speech synthesis is an active research area in speech synthesis. However, the intricate interplay of linguistic, emotional, and timbral features in cross-lingual scenarios poses significant challenges, often resulting in non-native accents and limited emotional expressiveness in generated speech. These issues are particularly pronounced in low-resource languages due to the scarcity of high-quality datasets and specialized modeling techniques. To address these challenges, we propose ZSEmo-MTVITS, a zero-shot cross-lingual emotional speech synthesis model for Mandarin and Tibetan based on the VITS. We constructed a novel Tibetan emotional speech dataset containing 2500 samples of the Lhasa dialect, covering five emotions (angry, happy, neutral, sad, and surprised), and evaluated Tibetan emotional dataset using the established end-to-end model VITS. Furthermore, we optimize the original VITS model by introducing extra emotion, language, speaker labels and a pre-trained timbre converter model to enable zero-shot cross-lingual emotional speech synthesis for Mandarin and Tibetan. The experimental results show that the ZSEmo-MTVITS model outperforms the baseline model in subjective evaluations, demonstrating superior emotional speech synthesis performance. The model effectively achieves Zero-Shot cross-lingual emotional speech synthesis for Mandarin and Tibetan.