Scholarlink

<p>Multimodal Sentiment Analysis is typically benchmarked on English text and image pairs with a single image, limiting the understanding of posts with multiple images, languages other than English, and spam. We introduce a spam-aware Multimodal Sentiment Analysis framework for Spanish X (Twitter) posts and report baselines on the Multimodal Spanish Sentiment Analysis Impact Dataset (MSSAID): 674 tweets about two Saúl “Canelo” Álvarez fights with up to four images each (804 total) and four labels (positive, negative, neutral, spam). The framework extracts text embeddings with a fine-tuned BETO model and visual embeddings with a fine-tuned Vision Transformer. For images containing text, a YOLOv8 detector localizes text regions, and EasyOCR extracts strings that are embedded in BETO. We compare two fusion strategies: sum fusion over [CLS] tokens and Transformer encoder fusion, followed by cost-sensitive Support Vector Machine classification under class imbalance. Ablations vary modality combinations and image counts, while stratified bootstrap testing estimates uncertainty for Matthews Correlation Coefficient differences. The best configuration uses tweet text plus only the first image with sum fusion, reaching 67.16% Matthews Correlation Coefficient (95% bootstrap confidence interval [0.5324, 0.8026]). Adding more images degrades results, and incorporating text extracted from images provides no reliable gains. Error analyzes show systematic confusion between neutral and spam. These findings provide a realistic Spanish benchmark and practical guidance on modality selection, fusion, embedded text, and spam handling.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

On the impact of image and text data on multimodal sentiment analysis in Spanish

  • 期刊论文

  • 发表日期：

    2026年3月16日

查看全文
  • Luis N. Zúñiga-Morales,
  • Jorge Ángel González-Ordiano,
  • J. Emilio Quiroz-Ibarra,
  • Steven J. Simske

摘要

Multimodal Sentiment Analysis is typically benchmarked on English text and image pairs with a single image, limiting the understanding of posts with multiple images, languages other than English, and spam. We introduce a spam-aware Multimodal Sentiment Analysis framework for Spanish X (Twitter) posts and report baselines on the Multimodal Spanish Sentiment Analysis Impact Dataset (MSSAID): 674 tweets about two Saúl “Canelo” Álvarez fights with up to four images each (804 total) and four labels (positive, negative, neutral, spam). The framework extracts text embeddings with a fine-tuned BETO model and visual embeddings with a fine-tuned Vision Transformer. For images containing text, a YOLOv8 detector localizes text regions, and EasyOCR extracts strings that are embedded in BETO. We compare two fusion strategies: sum fusion over [CLS] tokens and Transformer encoder fusion, followed by cost-sensitive Support Vector Machine classification under class imbalance. Ablations vary modality combinations and image counts, while stratified bootstrap testing estimates uncertainty for Matthews Correlation Coefficient differences. The best configuration uses tweet text plus only the first image with sum fusion, reaching 67.16% Matthews Correlation Coefficient (95% bootstrap confidence interval [0.5324, 0.8026]). Adding more images degrades results, and incorporating text extracted from images provides no reliable gains. Error analyzes show systematic confusion between neutral and spam. These findings provide a realistic Spanish benchmark and practical guidance on modality selection, fusion, embedded text, and spam handling.