Scholarlink

<p>Audio-driven video generation has recently shown significant progress in producing realistic visual content. However, most previous studies focus primarily on speech or other narrow-band audio sources, limiting their effectiveness in scenarios involving complex auditory inputs, such as musical performances that combine transient sounds with sustained tones. To overcome these constraints, we introduce SyncDiT, a novel framework designed for synchronized audio-visual generation from complex audio signals and reference images. At its core, SyncDiT leverages an audio encoder to capture intricate acoustic characteristics within an audio clip, while an Audio Sync Cross-Attention module aligns video latents with frame-level audio synchronization features. Additionally, we construct the Musical Instrument Video (MIV) dataset, comprising diverse high-resolution instrument performance videos to support this task. Extensive experiments demonstrate that SyncDiT achieves superior audio-visual synchronization and enhanced visual quality compared to existing approaches, underscoring its effectiveness in generating lifelike videos from complex audio inputs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SyncDIT: audio-visual aligned video generation with audio synchronization feature

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月19日

查看全文
  • Quanyue Song,
  • Zhizhi Guo,
  • Yishan He,
  • Zhihao Wang,
  • Zhixiang He,
  • Chi Zhang,
  • Caigui Jiang,
  • Xuelong Li

摘要

Audio-driven video generation has recently shown significant progress in producing realistic visual content. However, most previous studies focus primarily on speech or other narrow-band audio sources, limiting their effectiveness in scenarios involving complex auditory inputs, such as musical performances that combine transient sounds with sustained tones. To overcome these constraints, we introduce SyncDiT, a novel framework designed for synchronized audio-visual generation from complex audio signals and reference images. At its core, SyncDiT leverages an audio encoder to capture intricate acoustic characteristics within an audio clip, while an Audio Sync Cross-Attention module aligns video latents with frame-level audio synchronization features. Additionally, we construct the Musical Instrument Video (MIV) dataset, comprising diverse high-resolution instrument performance videos to support this task. Extensive experiments demonstrate that SyncDiT achieves superior audio-visual synchronization and enhanced visual quality compared to existing approaches, underscoring its effectiveness in generating lifelike videos from complex audio inputs.