Emotion and sentiment understanding in conversational settings still poses challenges with the intricate interaction of linguistic and non-linguistic cues in several modalities. Although unimodal methods based on pretrained models such as BERT for text or 3D CNNs for videos have shown good domain-specific accuracy, they do miss important cross-modality relationships. A favorable example includes the cases where sarcasm in the form of vocal intensity or a face that amplifies textual sentiment may be lost with single-modality systems. Multimodal fusion methods that have been proposed have two primary weaknesses: modality misalignment due to the sparse or noisy nature of available annotations in MELD and IEMOCAP and the extreme class imbalance that caused models to lean heavily in favor of majority-based emotion classes such as neutral and joy and underperform on minor classes. To address these challenges, we present a multimodal fusion framework to concurrently fuse three modalities with custom encoders: a frozen BERT encoder to preserve linguistic context with fewer trainable parameters, a 3D ResNet-18 video encoder to learn spatiotemporal face dynamics, and a 1D CNN audio encoder to encode prosodic characteristics with pitch and intensity variations. Our late fusion architecture projects each modality into a homogeneous 128-dimensional latent representation, which enables effective learning across modalities without learning explicit modulator-based alignment. We follow a dual-task learning framework with class-weighted cross-entropy loss and label smoothing to jointly learn seven-class emotion and three-class sentiment in parallel.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ConvEmoSentNet: A Parameter-Efficient Framework for Multimodal Emotion and Sentiment Analysis in Social Media Conversations

  • Akshay Sinha,
  • Gauri Saksena,
  • Yash Chandel

摘要

Emotion and sentiment understanding in conversational settings still poses challenges with the intricate interaction of linguistic and non-linguistic cues in several modalities. Although unimodal methods based on pretrained models such as BERT for text or 3D CNNs for videos have shown good domain-specific accuracy, they do miss important cross-modality relationships. A favorable example includes the cases where sarcasm in the form of vocal intensity or a face that amplifies textual sentiment may be lost with single-modality systems. Multimodal fusion methods that have been proposed have two primary weaknesses: modality misalignment due to the sparse or noisy nature of available annotations in MELD and IEMOCAP and the extreme class imbalance that caused models to lean heavily in favor of majority-based emotion classes such as neutral and joy and underperform on minor classes. To address these challenges, we present a multimodal fusion framework to concurrently fuse three modalities with custom encoders: a frozen BERT encoder to preserve linguistic context with fewer trainable parameters, a 3D ResNet-18 video encoder to learn spatiotemporal face dynamics, and a 1D CNN audio encoder to encode prosodic characteristics with pitch and intensity variations. Our late fusion architecture projects each modality into a homogeneous 128-dimensional latent representation, which enables effective learning across modalities without learning explicit modulator-based alignment. We follow a dual-task learning framework with class-weighted cross-entropy loss and label smoothing to jointly learn seven-class emotion and three-class sentiment in parallel.