One-shot talking head generation aims to produce a video that is consistent with the driving audio from a single frame of the source character. Recently, NeRF-based methods have made significant progress in novel view synthesis. However, these methods often generate unnatural or degraded images when applied to dynamic objects, primarily due to the loss of geometric consistency during the generation process. To address this issue, we propose a depth-guided keypoint warp mechanism that provides geometric guidance for NeRF rendering. Additionally, by combining a multi-layer perceptron (MLP) with an attention mechanism, which can encode more neighborhood information while reducing the total number of parameters. Finally, we design an adaptive loss function to minimize floaters and enhance lip-sync consistency. By integrating these methods, we introduce the DA-NeRF, which generates coherent and realistic talking head videos by accurately depicting lip motion, facial expressions, and head poses. Experimental results from several benchmark tests demonstrate that our method produces more realistic talking head videos, surpassing the state-of-the-art methods in image quality.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DA-NeRF: High-Fidelity Talking Face Generation From Speech With Neural Radiance Fields

  • Yali Cai,
  • Peng Qiao,
  • Dongsheng Li

摘要

One-shot talking head generation aims to produce a video that is consistent with the driving audio from a single frame of the source character. Recently, NeRF-based methods have made significant progress in novel view synthesis. However, these methods often generate unnatural or degraded images when applied to dynamic objects, primarily due to the loss of geometric consistency during the generation process. To address this issue, we propose a depth-guided keypoint warp mechanism that provides geometric guidance for NeRF rendering. Additionally, by combining a multi-layer perceptron (MLP) with an attention mechanism, which can encode more neighborhood information while reducing the total number of parameters. Finally, we design an adaptive loss function to minimize floaters and enhance lip-sync consistency. By integrating these methods, we introduce the DA-NeRF, which generates coherent and realistic talking head videos by accurately depicting lip motion, facial expressions, and head poses. Experimental results from several benchmark tests demonstrate that our method produces more realistic talking head videos, surpassing the state-of-the-art methods in image quality.