Scholarlink

<p>Emotion recognition plays a crucial role in human-computer interaction, sentiment analysis, and affective computing, enabling more natural and intuitive communication between machines and users. Although multimodal approaches that leverage both acoustic and textual cues offer a more comprehensive understanding of emotional states, they remain challenging owing to the complexity of human emotions, modality-specific variations, and class imbalance in emotional datasets. Recent transformer-based models have advanced feature extraction in the speech and text domains. However, many existing methods rely on complex fusion architectures or struggle with underrepresented emotions. To address these challenges, we propose a probability-level multimodal fusion framework that integrates a Vision Transformer (ViT) trained on spectrogram-based acoustic representations and Bidirectional Encoder Representations from Transformers (BERT) for textual modeling. The modality-wise emotion posterior probabilities were combined using a lightweight fusion classifier to exploit complementary confidence patterns without feature-level entanglement. Additionally, we introduced a proportional augmentation technique to mitigate class imbalances, ensuring a fairer representation of minority emotions without compromising data integrity. Evaluated on a benchmark dataset for conversational emotion recognition, the proposed approach consistently outperformed unimodal baselines and yielded competitive performance compared with recent multimodal methods, particularly in terms of weighted recall and F1-score for underrepresented emotions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Transformer-based fusion of acoustic and textual cues with proportional augmentation for emotion recognition

  • 期刊论文

  • 发表日期：

    2026年2月23日

查看全文
  • Shintami Chusnul Hidayati,
  • Khaela Fortunela

摘要

Emotion recognition plays a crucial role in human-computer interaction, sentiment analysis, and affective computing, enabling more natural and intuitive communication between machines and users. Although multimodal approaches that leverage both acoustic and textual cues offer a more comprehensive understanding of emotional states, they remain challenging owing to the complexity of human emotions, modality-specific variations, and class imbalance in emotional datasets. Recent transformer-based models have advanced feature extraction in the speech and text domains. However, many existing methods rely on complex fusion architectures or struggle with underrepresented emotions. To address these challenges, we propose a probability-level multimodal fusion framework that integrates a Vision Transformer (ViT) trained on spectrogram-based acoustic representations and Bidirectional Encoder Representations from Transformers (BERT) for textual modeling. The modality-wise emotion posterior probabilities were combined using a lightweight fusion classifier to exploit complementary confidence patterns without feature-level entanglement. Additionally, we introduced a proportional augmentation technique to mitigate class imbalances, ensuring a fairer representation of minority emotions without compromising data integrity. Evaluated on a benchmark dataset for conversational emotion recognition, the proposed approach consistently outperformed unimodal baselines and yielded competitive performance compared with recent multimodal methods, particularly in terms of weighted recall and F1-score for underrepresented emotions.