Deep multimodal networks for detecting disaster-related informative content on social media
摘要
In times of natural disasters and crises, social media sites are crucial sources of real-time information. Yet, the sheer volume and multimodal nature of this data—encompassing both text and images—make it difficult to quickly identify relevant and actionable content. This study presents a multimodal deep learning model that integrates Bidirectional Encoder Representations from Transformers (BERT) for text and Vision Transformer (ViT) for images. A fusion mechanism is used to combine both modalities for more accurate classification of posts as informative or non-informative. The proposed model is evaluated on benchmark crisis datasets with extensive preprocessing, data augmentation, and regularization strategies to ensure robustness across various disaster scenarios. Experimental results demonstrate that the multimodal approach significantly outperforms unimodal models. The BERT-based text-only model achieves an F1 score of 0.82, the ViT-based image-only model reaches 0.87, while the multimodal BERT+ViT model achieves the highest F1 score of 0.89. Furthermore, it attains an AUC of 0.97 and a validation accuracy of 0.91, confirming the advantage of integrating textual and visual features for real-time disaster informativeness detection. These results highlight the model’s potential to assist emergency responders in filtering high-priority content during critical situations.