Emotion-Aware Speech-Driven Facial Avatar Animation via Joint Blendshape Prediction and Emotion Recognition
摘要
The evolution of virtual and augmented reality technologies, accompanied by advances in animation and gaming industries demand novel advances for facial avatar generation. Exiting methods offer diverse approaches in terms of realism, expressiveness, and quality. However, limitations persist in achieving real-time responsiveness and integration with standard animation frameworks. This paper introduced a series of emotion-aware facial animation models designed for both offline and real-time applications. The proposed models generate expressive facial animations by jointly predicting ARKit blendshape coefficients and the emotion conveyed in the speech signal. The proposed models rely on a two-stage architecture comprising a frozen WavLM-based upstream network and a trainable downstream module. A multitask learning strategy is adopted to jointly optimise blendshape regression and emotion recognition. Objective evaluations demonstrated that the proposed models outperformed baseline systems both in blendshape prediction and emotion classification tasks. A user study was also conducted to evaluate the perceived expressiveness, coherence, and quality of the generated animations, confirming the effectiveness of the models. Finally, run-time performance was assessed under realistic streaming conditions, highlighting the potential to deliver animations with low latency and resource usage.