Scholarlink

<p>Voice cloning requires both naturalness and balanced efficiency. However, single-task models often entangle timbre and prosody, limiting controllability and affecting generation speed. This study introduces Multi-Task Contrastive Voice Cloning (MTC-VC), a multi-task contrastive method that explicitly decouples timbre, speaking rate, and emotion to enhance controllability, achieve a balanced quality–speed trade-off, and improve expressiveness. A three-branch encoder is adopted: a contrastive branch enhances timbre discriminability, and two auxiliary branches model speaking rate and emotion. Learnable fusion weights adaptively combine the branch representations, while discrete rate and emotion labels provide explicit control at inference; the fused representation then conditions a flow-based synthesizer. Trained on LibriSpeech (English), the evaluation uses one-shot prompts at inference spanning English and Mandarin accents, including a Mandarin-accented cartoon-style male, and reports ten-speaker macro-averages; under GPU, MTC-VC achieves an 8.74% lower MCD, a 3.62% higher SIM, and a 10.29% higher SSR than OpenVoice V2, while targeting a balanced quality–speed operating point. Mean Opinion Scores (95% confidence intervals) favor MTC-VC, reaching 4.18 for naturalness, 4.13 for speaker similarity, and 4.12 for emotion appropriateness. Across nine English targets and one Mandarin-accented target, the cross-system ranking is preserved with only small between-speaker deviations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MTC-VC: A Multi-Task Contrastive Learning Method for Controllable and Efficiency-Balanced Voice Cloning

  • 期刊论文

  • 发表日期：

    2026年2月24日

查看全文
  • Rui Zhou,
  • Yan Zhu,
  • Baili Zhang

摘要

Voice cloning requires both naturalness and balanced efficiency. However, single-task models often entangle timbre and prosody, limiting controllability and affecting generation speed. This study introduces Multi-Task Contrastive Voice Cloning (MTC-VC), a multi-task contrastive method that explicitly decouples timbre, speaking rate, and emotion to enhance controllability, achieve a balanced quality–speed trade-off, and improve expressiveness. A three-branch encoder is adopted: a contrastive branch enhances timbre discriminability, and two auxiliary branches model speaking rate and emotion. Learnable fusion weights adaptively combine the branch representations, while discrete rate and emotion labels provide explicit control at inference; the fused representation then conditions a flow-based synthesizer. Trained on LibriSpeech (English), the evaluation uses one-shot prompts at inference spanning English and Mandarin accents, including a Mandarin-accented cartoon-style male, and reports ten-speaker macro-averages; under GPU, MTC-VC achieves an 8.74% lower MCD, a 3.62% higher SIM, and a 10.29% higher SSR than OpenVoice V2, while targeting a balanced quality–speed operating point. Mean Opinion Scores (95% confidence intervals) favor MTC-VC, reaching 4.18 for naturalness, 4.13 for speaker similarity, and 4.12 for emotion appropriateness. Across nine English targets and one Mandarin-accented target, the cross-system ranking is preserved with only small between-speaker deviations.