Scholarlink

<p>Speech Emotion Recognition (SER) plays a major role in enhancing human–computer interaction and affective computing but still unimodal methods often struggle with poor generalization and robustness in noisy and cross-domain scenarios in SER. To address this limitation, this paper introduces a multimodal SER system that combines both acoustic and linguistic modalities for richer emotional representations. The acoustic feature extraction captures the prosodic, spectral, and deep acoustic representations yielded by the Multi-Layer Emotion Fusion Transformer (MLEFT). In contrast, the linguistic features are extracted from a Generative Pre-trained Transformer-2(GPT-2) model based on Whisper-transcribed transcripts. The features are then combined using a new cross-attention mechanism, followed by a Bi-directional Gated Recurrent Unit (BiGRU) and multi-head attention layer to capture temporal dependencies and rich cross-modal interactions. Empirical evaluation was conducted across a wide range of datasets, including RAVDESS, SAVEE, TESS, IEMOCAP, MELD, and in-house English and Hindi corpora. Experimental results show that the holistic fusion of prosodic, spectral, and deep acoustic features with contextual linguistic embeddings significantly enhances the recognition performance. Notably, the present cross-attention fusion approach, coupled with BiGRU, multi-head attention, and FCN classification pipeline, repeatedly produced state-of-the-art weighted and unweighted accuracies. The results confirm the excellence of the framework’s performance and its potential for stable, multilingual SER applications in complex real-world settings.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MLEFT-LF: Leveraging Multi-layer Emotion Fusion Transformer and GPT-2 Linguistic Features with Cross-Attention for Speech Emotion Recognition

  • 期刊论文

  • 发表日期：

    2026年6月14日

查看全文
  • Syed Shareefunnisa,
  • Venkata Krishna Kishore Kolli

摘要

Speech Emotion Recognition (SER) plays a major role in enhancing human–computer interaction and affective computing but still unimodal methods often struggle with poor generalization and robustness in noisy and cross-domain scenarios in SER. To address this limitation, this paper introduces a multimodal SER system that combines both acoustic and linguistic modalities for richer emotional representations. The acoustic feature extraction captures the prosodic, spectral, and deep acoustic representations yielded by the Multi-Layer Emotion Fusion Transformer (MLEFT). In contrast, the linguistic features are extracted from a Generative Pre-trained Transformer-2(GPT-2) model based on Whisper-transcribed transcripts. The features are then combined using a new cross-attention mechanism, followed by a Bi-directional Gated Recurrent Unit (BiGRU) and multi-head attention layer to capture temporal dependencies and rich cross-modal interactions. Empirical evaluation was conducted across a wide range of datasets, including RAVDESS, SAVEE, TESS, IEMOCAP, MELD, and in-house English and Hindi corpora. Experimental results show that the holistic fusion of prosodic, spectral, and deep acoustic features with contextual linguistic embeddings significantly enhances the recognition performance. Notably, the present cross-attention fusion approach, coupled with BiGRU, multi-head attention, and FCN classification pipeline, repeatedly produced state-of-the-art weighted and unweighted accuracies. The results confirm the excellence of the framework’s performance and its potential for stable, multilingual SER applications in complex real-world settings.