Multi-view Interaction Network with Guided Contrastive Learning for Multimodal Summarization
摘要
Unlike traditional unimodal summarization tasks, multimodal summarization with multimodal output (MSMO) task aims to align the semantic information between modalities to generate more accurate multimodal summaries for lengthy representation sequences. However, existing methods either overlook the potential semantic correlations between multimodal information or fail to achieve fine-grained interaction learning between modalities, which hinders performance improvement. In view of this, we propose a Multi-view Interaction Network with Guided Contrastive Learning (MINGCL) to address the MSMO. We first build a multimodal feature interaction component spanning multiple views, from events to entities, to fully explore the potential semantic consistency between modalities. Furthermore, we employ contrastive learning to train and optimize the model, incorporating a guided module to aid in generating high-quality positive and negative samples, thereby enhancing the model’s ability to distinguish summary content from other information. Finally, we conduct extensive experiments on two video summarization datasets (SumMe and TVSum) and three multimodal summarization datasets (CNN, Daily Mail, and BLiSS). Our model achieves state-of-the-art results.