mmWave-based Facial Expression Reconstruction
摘要
In recent times, there has been a rapid rise in the market for facial expression reconstruction technologies, which enable a wide range of face-driven applications, such as VR modeling, human-computer interaction, and affective computing. Existing methods for 3D facial reconstruction rely heavily on cameras, which can raise privacy concerns and struggle in scenarios with obstructions or poor lighting. In this chapter, we introduce a passive and privacy-conscious 3D facial expression reconstruction framework, mm3DFace, which utilizes a mmWave radar to reconstruct dynamic facial expressions. mm3DFace processes the pre-captured mmWave radar signals and extracts key facial geometric features that reflect subtle facial changes using a ConvNeXt model with triple loss embedding. Following this, mm3DFace derives robust facial shapes that are invariant to distance and orientation leveraging 68 facial landmarks through a region-divided affine transformation. Subsequently, mm3DFace enhances the reconstructed facial expressions by applying a region-based amplification technique and then generates 3D avatars that represent the facial expressions. Evaluations in real-world environments involving 15 participants demonstrate that mm3DFace can accurately track 68 facial landmarks with an average normalized mean error of 3.94%, a mean absolute error of \(2.30\) mm, and a 3D mean absolute error of \(4.10\) mm. These results validate the effectiveness and practicality of the system for real-world facial expression reconstruction applications.