Information fragmentation is a defining property of social media: early posts are short, informal, and often omit crucial actors, locations, or explicit event keywords. In this chapter, we present Multimodal Fusion with External Knowledge (MFEK), a framework that treats missing context as a first-class bottleneck and recovers it by integrating external knowledge with multimodal evidence. We first analyze why fragmentation causes predictable failures such as missed early signals, wrong grounding, and shallow clustering. We then introduce the MFEK architecture as a dual-source knowledge integration pipeline, including multimodal information extraction, knowledge retrieval from complementary sources, and knowledge-aware fusion via attention-based contextualization. Next, we present the Social Event Detection (SED) dataset, describing its collection, annotation, and key statistics designed to reflect realistic context scarcity. Finally, we report extensive experiments, including strong baselines and ablation analyses, to quantify when knowledge improves detection robustness and clustering stability and when noisy knowledge can hurt, offering practical guidance for deployment under real-time constraints.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Fusion with External Knowledge (MFEK)

  • Zehang Lin,
  • Qing Li

摘要

Information fragmentation is a defining property of social media: early posts are short, informal, and often omit crucial actors, locations, or explicit event keywords. In this chapter, we present Multimodal Fusion with External Knowledge (MFEK), a framework that treats missing context as a first-class bottleneck and recovers it by integrating external knowledge with multimodal evidence. We first analyze why fragmentation causes predictable failures such as missed early signals, wrong grounding, and shallow clustering. We then introduce the MFEK architecture as a dual-source knowledge integration pipeline, including multimodal information extraction, knowledge retrieval from complementary sources, and knowledge-aware fusion via attention-based contextualization. Next, we present the Social Event Detection (SED) dataset, describing its collection, annotation, and key statistics designed to reflect realistic context scarcity. Finally, we report extensive experiments, including strong baselines and ablation analyses, to quantify when knowledge improves detection robustness and clustering stability and when noisy knowledge can hurt, offering practical guidance for deployment under real-time constraints.