Scholarlink

<p>In music emotion recognition (MER), capturing fine-grained interactions among multimodal cues—particularly acoustic signals and lyrical content—remains challenging due to their intrinsic structural complexity and temporal dependencies. In this study, we propose GEMMER (Graph-Enhanced Multi-modal Music Emotion Recognition), a novel graph-enhanced multimodal framework designed to explicitly encode and leverage salient structural information inherent in musical content. Specifically, GEMMER introduces the concept of Salient Musical Units (SMUs) as dynamically identified emotional anchors, using graph neural networks (GNNs) to model both inter- and intra-SMU dependencies effectively. Specifically, multimodal fusion is achieved via a hierarchical, context-aware Cross-modal Fusion Core (CFC), which utilizes multi-head attention to deeply integrate acoustic and lyrical representations. Subsequently, dual-graph GNN modules (i.e., inter-SMU interaction graph and intra-SMU evolution graph) explicitly model emotional progressions and interactions among SMUs. To ensure structural robustness and diversity, we incorporate orthogonality regularization and diversity-enhanced SMU embeddings into the joint optimization strategy. Extensive experiments conducted on two benchmark datasets, PMEmo and DEAM, demonstrate GEMMER’s effectiveness: achieving segment-level accuracy and weighted F1-scores of 85.4% and 84.7% on PMEmo, surpassing state-of-the-art baselines by at least 2.3%; and significantly outperforming comparative approaches with an accuracy improvement of up to 2.7% on DEAM. Additionally, GEMMER attains a 91.3% quadrant match rate for piece-level emotion prediction, highlighting its superior capacity to capture global emotional coherence. Our framework also strikes an optimal balance between computational efficiency (43.6&#xa0;M parameters, ~ 11.6&#xa0;ms per inference) and predictive accuracy, affirming its practical applicability for real-world scenarios, supported by rigorous robustness analyses against labeling noise.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Salient structural modeling in multimodal music emotion recognition via graph neural networks

  • 期刊论文

  • 发表日期：

    2026年3月10日

查看全文
  • Yan Wu,
  • Wei Hu

摘要

In music emotion recognition (MER), capturing fine-grained interactions among multimodal cues—particularly acoustic signals and lyrical content—remains challenging due to their intrinsic structural complexity and temporal dependencies. In this study, we propose GEMMER (Graph-Enhanced Multi-modal Music Emotion Recognition), a novel graph-enhanced multimodal framework designed to explicitly encode and leverage salient structural information inherent in musical content. Specifically, GEMMER introduces the concept of Salient Musical Units (SMUs) as dynamically identified emotional anchors, using graph neural networks (GNNs) to model both inter- and intra-SMU dependencies effectively. Specifically, multimodal fusion is achieved via a hierarchical, context-aware Cross-modal Fusion Core (CFC), which utilizes multi-head attention to deeply integrate acoustic and lyrical representations. Subsequently, dual-graph GNN modules (i.e., inter-SMU interaction graph and intra-SMU evolution graph) explicitly model emotional progressions and interactions among SMUs. To ensure structural robustness and diversity, we incorporate orthogonality regularization and diversity-enhanced SMU embeddings into the joint optimization strategy. Extensive experiments conducted on two benchmark datasets, PMEmo and DEAM, demonstrate GEMMER’s effectiveness: achieving segment-level accuracy and weighted F1-scores of 85.4% and 84.7% on PMEmo, surpassing state-of-the-art baselines by at least 2.3%; and significantly outperforming comparative approaches with an accuracy improvement of up to 2.7% on DEAM. Additionally, GEMMER attains a 91.3% quadrant match rate for piece-level emotion prediction, highlighting its superior capacity to capture global emotional coherence. Our framework also strikes an optimal balance between computational efficiency (43.6 M parameters, ~ 11.6 ms per inference) and predictive accuracy, affirming its practical applicability for real-world scenarios, supported by rigorous robustness analyses against labeling noise.