The evolution of virtual and augmented reality technologies, accompanied by advances in animation and gaming industries demand novel advances for facial avatar generation. Exiting methods offer diverse approaches in terms of realism, expressiveness, and quality. However, limitations persist in achieving real-time responsiveness and integration with standard animation frameworks. This paper introduced a series of emotion-aware facial animation models designed for both offline and real-time applications. The proposed models generate expressive facial animations by jointly predicting ARKit blendshape coefficients and the emotion conveyed in the speech signal. The proposed models rely on a two-stage architecture comprising a frozen WavLM-based upstream network and a trainable downstream module. A multitask learning strategy is adopted to jointly optimise blendshape regression and emotion recognition. Objective evaluations demonstrated that the proposed models outperformed baseline systems both in blendshape prediction and emotion classification tasks. A user study was also conducted to evaluate the perceived expressiveness, coherence, and quality of the generated animations, confirming the effectiveness of the models. Finally, run-time performance was assessed under realistic streaming conditions, highlighting the potential to deliver animations with low latency and resource usage.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Emotion-Aware Speech-Driven Facial Avatar Animation via Joint Blendshape Prediction and Emotion Recognition

  • Juan Camilo Vásquez-Correa,
  • Ander González-Docasal,
  • Aritz Lasarguren,
  • Jone López,
  • Aitor Álvarez,
  • Egoitz Rodriguez-Olea

摘要

The evolution of virtual and augmented reality technologies, accompanied by advances in animation and gaming industries demand novel advances for facial avatar generation. Exiting methods offer diverse approaches in terms of realism, expressiveness, and quality. However, limitations persist in achieving real-time responsiveness and integration with standard animation frameworks. This paper introduced a series of emotion-aware facial animation models designed for both offline and real-time applications. The proposed models generate expressive facial animations by jointly predicting ARKit blendshape coefficients and the emotion conveyed in the speech signal. The proposed models rely on a two-stage architecture comprising a frozen WavLM-based upstream network and a trainable downstream module. A multitask learning strategy is adopted to jointly optimise blendshape regression and emotion recognition. Objective evaluations demonstrated that the proposed models outperformed baseline systems both in blendshape prediction and emotion classification tasks. A user study was also conducted to evaluate the perceived expressiveness, coherence, and quality of the generated animations, confirming the effectiveness of the models. Finally, run-time performance was assessed under realistic streaming conditions, highlighting the potential to deliver animations with low latency and resource usage.