What do we learn from a single frame of a GIF or a short clip? A fleeting snapshot may capture a smirk, but it rarely conveys the full spectrum of human emotion. Understanding emotional transitions over time is crucial for a complete analysis. This work introduces a dynamic emotion recognition framework that integrates FER-VIT, a fine-tuned Visual Transformer, with FER-GPT, a prompt-engineered language model, to generate natural language descriptions of emotional changes. Experiments on the CREMA-D dataset show that FER-VIT effectively classifies emotions, particularly distinct ones like Happiness and Anger, while FER-GPT enhances interpretability by translating raw emotion scores into descriptive captions. This system has broad applications in mental health monitoring, human-computer interaction, and sentiment analysis, enabling AI to better understand and respond to human emotions. By bridging vision and language, our approach provides a more intuitive way to interpret and analyze dynamic emotional expressions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Framework for Emotional Transition Detection

  • Pierre Jecrois,
  • Salem Othman,
  • Leonidas Deligiannidis,
  • Yetunde Folajimi

摘要

What do we learn from a single frame of a GIF or a short clip? A fleeting snapshot may capture a smirk, but it rarely conveys the full spectrum of human emotion. Understanding emotional transitions over time is crucial for a complete analysis. This work introduces a dynamic emotion recognition framework that integrates FER-VIT, a fine-tuned Visual Transformer, with FER-GPT, a prompt-engineered language model, to generate natural language descriptions of emotional changes. Experiments on the CREMA-D dataset show that FER-VIT effectively classifies emotions, particularly distinct ones like Happiness and Anger, while FER-GPT enhances interpretability by translating raw emotion scores into descriptive captions. This system has broad applications in mental health monitoring, human-computer interaction, and sentiment analysis, enabling AI to better understand and respond to human emotions. By bridging vision and language, our approach provides a more intuitive way to interpret and analyze dynamic emotional expressions.