DA-NeRF: High-Fidelity Talking Face Generation From Speech With Neural Radiance Fields
摘要
One-shot talking head generation aims to produce a video that is consistent with the driving audio from a single frame of the source character. Recently, NeRF-based methods have made significant progress in novel view synthesis. However, these methods often generate unnatural or degraded images when applied to dynamic objects, primarily due to the loss of geometric consistency during the generation process. To address this issue, we propose a depth-guided keypoint warp mechanism that provides geometric guidance for NeRF rendering. Additionally, by combining a multi-layer perceptron (MLP) with an attention mechanism, which can encode more neighborhood information while reducing the total number of parameters. Finally, we design an adaptive loss function to minimize floaters and enhance lip-sync consistency. By integrating these methods, we introduce the DA-NeRF, which generates coherent and realistic talking head videos by accurately depicting lip motion, facial expressions, and head poses. Experimental results from several benchmark tests demonstrate that our method produces more realistic talking head videos, surpassing the state-of-the-art methods in image quality.