Multimodal content (text and images) that offers situational insights has emerged as a critical source for real-time disaster response on social media platforms. Nevertheless, the automated analysis of this data is significantly impeded by its unstructured and chaotic nature. This study introduces a multimodal classification framework that leverages CLIP (Contrastive Language–Image Pretraining) to extract semantically rich embeddings from tweet text and images for multiclass humanitarian categorization across eight categories. We evaluate two fusion strategies on the CrisisMMD v2.0 dataset: a baseline trainable Multi-Layer Perceptron (MLP) fusion head to model non-linear interactions, and a Cross-Modal Attention mechanism that enables fine-grained alignment between modalities via multi-head attention. Both methods incorporate cosine similarity as a global consistency feature. The Cross-Modal Attention approach achieved the highest accuracy and F1 score, especially in managing inter-modal inconsistencies, as evidenced by experimental results showing that the trainable fusion models outperform simpler baselines. These findings demonstrate the effectiveness of deep fusion strategies for robust disaster classification, especially in the presence of noisy or ambiguous social media content.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CLIP-Based Multimodal Disaster Classification: MLP vs Cross-Modal Attention Fusion

  • Sireesha Vedururu,
  • Chandrasekhar Uddagiri,
  • Pavan Kumar Reddy Yekollu,
  • Sreelakshmi Doma

摘要

Multimodal content (text and images) that offers situational insights has emerged as a critical source for real-time disaster response on social media platforms. Nevertheless, the automated analysis of this data is significantly impeded by its unstructured and chaotic nature. This study introduces a multimodal classification framework that leverages CLIP (Contrastive Language–Image Pretraining) to extract semantically rich embeddings from tweet text and images for multiclass humanitarian categorization across eight categories. We evaluate two fusion strategies on the CrisisMMD v2.0 dataset: a baseline trainable Multi-Layer Perceptron (MLP) fusion head to model non-linear interactions, and a Cross-Modal Attention mechanism that enables fine-grained alignment between modalities via multi-head attention. Both methods incorporate cosine similarity as a global consistency feature. The Cross-Modal Attention approach achieved the highest accuracy and F1 score, especially in managing inter-modal inconsistencies, as evidenced by experimental results showing that the trainable fusion models outperform simpler baselines. These findings demonstrate the effectiveness of deep fusion strategies for robust disaster classification, especially in the presence of noisy or ambiguous social media content.