Scholarlink

The Latent Diffusion Model has been effectively applied in text-to-audio (T2A) synthesis tasks. However, several challenges impede its broader implementation, including the lack of audio-text paired data, low sensitivity to temporal words, and slow inference speeds. To address these issues, we employ an approach to convert the text into different expressions and introduce a new temporal encoder that directs the model to focus more on the sequence of actions. Meanwhile we prune the model to simplify the structure at each step, thereby enhancing inference speed. To prevent a decline in generation quality due to the reduction of replication computations for deep-level features, we add a channel-enhanced and spatial attention mechanism in the last two layers of the U-Net architecture to effectively extract features. Our model, trained on a single A100 GPU with the AudioCaps dataset, achieves a 73% improvement in inference speed compared to the baseline model, while also improving the quality of the generated audio.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Latent Diffusion Models for Text-to-Audio Generation with Limited Training Data

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Haiyan Jin,
  • Jiahong Zhu,
  • Tianlong Yang,
  • Hongwei Zuo,
  • Wei Wang

摘要

The Latent Diffusion Model has been effectively applied in text-to-audio (T2A) synthesis tasks. However, several challenges impede its broader implementation, including the lack of audio-text paired data, low sensitivity to temporal words, and slow inference speeds. To address these issues, we employ an approach to convert the text into different expressions and introduce a new temporal encoder that directs the model to focus more on the sequence of actions. Meanwhile we prune the model to simplify the structure at each step, thereby enhancing inference speed. To prevent a decline in generation quality due to the reduction of replication computations for deep-level features, we add a channel-enhanced and spatial attention mechanism in the last two layers of the U-Net architecture to effectively extract features. Our model, trained on a single A100 GPU with the AudioCaps dataset, achieves a 73% improvement in inference speed compared to the baseline model, while also improving the quality of the generated audio.