REA-TTS: Retrieval-Augmented Expressive Audiobook Text-to-Speech Generation with Contrastive Language-Audio Learning
摘要
In recent years, speech synthesis technology can already synthesize sentence-level speech with high expressiveness based on reference speech. However, achieving highly natural, expressive audiobook speech synthesis remains a considerable challenge. To improve the expression of synthesized audiobook speech based on reference speech, we proposed retrieval-augmented expressive audiobook text-to-speech, REA-TTS, a high-expressivity speech synthesis method that rivals human speech in timbre, prosody, and emotional expression for long text synthesis. We adapted contrastive learning and retrieval-augmented generation to an end-to-end speech synthesis framework. The framework integrates emotion contrastive learning and reference audio retrieval. It aligns audio and text emotion embeddings into the same latent space. Then, it uses cosine similarity to retrieve the audio that corresponds to the text as reference audio. This process enhances the naturalness and expressiveness of audiobook speech synthesis. Furthermore, we constructed a concatenated reference speech process, which can improve prosody variation. Our proposed method outperforms baseline systems in both intonation naturalness and emotional expressivity, effectively improving the overall perceptual quality of synthesized speech.