Sync2Lip: Enhancing DINet for Improved LipSync Video Generation
摘要
Lip-sync in video generation has advanced through deep learning models, but approaches still face the issues of visual artifacts, and low user control over the content. These shortcomings lead to overall low effectiveness and user satisfaction. This work proposes Sync2Lip an extension of the DINet model to address these constraints. Specifically, we introduce alpha blending to enhance visual smoothness by adding a controlled level of original input frame, probably reducing observable artifacts. Besides, our extension has a user side filter size that can be adjusted, which is imposed on the mouth area, and which virtually suppresses box-like distortion in the region of interest, thus providing greater flexibility for customization. Experimentation on our approach, Sync2Lip, demonstrates quantifiably better performance in key criteria such as Normalized Mean Square Error (NMSE), with a 30.77% decrease as compared to DINet, and an increase of approximately 1.357% in the Deformation Quality Score (DQS). By adding more control over filtering in the mouth region and taking advantage of alpha blending, it provides a simple and efficient means of fine-tuning the existing GAN-based video generation models. This results in more natural, visually coherent lip-sync that is better aligned with input audio and video features. However, this approach could still be prone to challenges under circumstances where extremely unfavorable head poses, or significant occlusions exist.