In recent times, there has been a rapid rise in the market for facial expression reconstruction technologies, which enable a wide range of face-driven applications, such as VR modeling, human-computer interaction, and affective computing. Existing methods for 3D facial reconstruction rely heavily on cameras, which can raise privacy concerns and struggle in scenarios with obstructions or poor lighting. In this chapter, we introduce a passive and privacy-conscious 3D facial expression reconstruction framework, mm3DFace, which utilizes a mmWave radar to reconstruct dynamic facial expressions. mm3DFace processes the pre-captured mmWave radar signals and extracts key facial geometric features that reflect subtle facial changes using a ConvNeXt model with triple loss embedding. Following this, mm3DFace derives robust facial shapes that are invariant to distance and orientation leveraging 68 facial landmarks through a region-divided affine transformation. Subsequently, mm3DFace enhances the reconstructed facial expressions by applying a region-based amplification technique and then generates 3D avatars that represent the facial expressions. Evaluations in real-world environments involving 15 participants demonstrate that mm3DFace can accurately track 68 facial landmarks with an average normalized mean error of 3.94%, a mean absolute error of \(2.30\) mm, and a 3D mean absolute error of \(4.10\) mm. These results validate the effectiveness and practicality of the system for real-world facial expression reconstruction applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

mmWave-based Facial Expression Reconstruction

  • Hao Kong,
  • Jiadi Yu,
  • Xuemin Sherman Shen

摘要

In recent times, there has been a rapid rise in the market for facial expression reconstruction technologies, which enable a wide range of face-driven applications, such as VR modeling, human-computer interaction, and affective computing. Existing methods for 3D facial reconstruction rely heavily on cameras, which can raise privacy concerns and struggle in scenarios with obstructions or poor lighting. In this chapter, we introduce a passive and privacy-conscious 3D facial expression reconstruction framework, mm3DFace, which utilizes a mmWave radar to reconstruct dynamic facial expressions. mm3DFace processes the pre-captured mmWave radar signals and extracts key facial geometric features that reflect subtle facial changes using a ConvNeXt model with triple loss embedding. Following this, mm3DFace derives robust facial shapes that are invariant to distance and orientation leveraging 68 facial landmarks through a region-divided affine transformation. Subsequently, mm3DFace enhances the reconstructed facial expressions by applying a region-based amplification technique and then generates 3D avatars that represent the facial expressions. Evaluations in real-world environments involving 15 participants demonstrate that mm3DFace can accurately track 68 facial landmarks with an average normalized mean error of 3.94%, a mean absolute error of \(2.30\) mm, and a 3D mean absolute error of \(4.10\) mm. These results validate the effectiveness and practicality of the system for real-world facial expression reconstruction applications.