Scholarlink

<p>In the field of face generation, diffusion model-based generation tasks have achieved remarkable realism and detail expressiveness. However, the synchronization of audio and video remains a critical challenge to be addressed. To tackle the above problem, this paper presents a two-stage method called adaptive diffusion landmark dynamic rendering (DLDR) for generating realistic talking face videos where mouth movements are synchronized with the semantic content of audio. In the first stage, an audio semantic mapping module (ASM) is designed based on a diffusion model, utilizing a transformer for adaptive feature fusion conditional information to generate mouth shape landmarks from audio features. In the second stage, a dynamic rendering module (DR) is proposed, consisting of a landmark alignment network and a conversion network. This module fuses the generated facial landmarks with the target image to produce synchronized talking face videos. Extensive experiments on the LRS2 and LRW datasets demonstrate that our method significantly improves the realism and mouth shape synchronization of generated talking face videos compared to state-of-the-art methods, achieving a 10.51% improvement in mouth shape synchronization confidence and a 13.78% increase in peak signal-to-noise ratio. We can watch the project code: <a href="https://github.com/tang-ying09/DLDR/tree/master">https://github.com/tang-ying09/DLDR/tree/master</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive diffusion landmark dynamic rendering for realistic talking face video generation

  • 期刊论文

  • 发表日期：

    2025年5月4日

查看全文
  • Tang Ying,
  • Liu Yazhi,
  • Li Xiong,
  • Li Wei

摘要

In the field of face generation, diffusion model-based generation tasks have achieved remarkable realism and detail expressiveness. However, the synchronization of audio and video remains a critical challenge to be addressed. To tackle the above problem, this paper presents a two-stage method called adaptive diffusion landmark dynamic rendering (DLDR) for generating realistic talking face videos where mouth movements are synchronized with the semantic content of audio. In the first stage, an audio semantic mapping module (ASM) is designed based on a diffusion model, utilizing a transformer for adaptive feature fusion conditional information to generate mouth shape landmarks from audio features. In the second stage, a dynamic rendering module (DR) is proposed, consisting of a landmark alignment network and a conversion network. This module fuses the generated facial landmarks with the target image to produce synchronized talking face videos. Extensive experiments on the LRS2 and LRW datasets demonstrate that our method significantly improves the realism and mouth shape synchronization of generated talking face videos compared to state-of-the-art methods, achieving a 10.51% improvement in mouth shape synchronization confidence and a 13.78% increase in peak signal-to-noise ratio. We can watch the project code: https://github.com/tang-ying09/DLDR/tree/master.