Elemental Discourse Unit Guidance Based Model for Multimodal Sentence Summarization
摘要
Multimodal sentence summarization (MMSS) intends to compress an original long sentence into a concise short sentence summary by leveraging textual and visual information. One of the primary objectives in contemporary multimodal sentence summarization tasks is the extraction of crucial information from the multimodal content, which then guides the generation of accurate and coherent summaries. Existing methods often employ implicit learning to identify key information, but this approach lacks transparency and interpretability, making it difficult to determine whether the crucial information has been effectively learned. Consequently, recent attention has shifted towards explicit key information learning methods, such as tokens. However, tokens often provide incomplete semantic information and may have ambiguity. To address these issues, we propose EDU-MMSS, an Elemental Discourse Unit (EDU) guidance based model for MultiModal Sentence Summarisation. Compared to tokens, EDU is considered independent and complete atomic units of semantic information, making them more suitable for expressing both textual and visual information simultaneously. Our innovation is two-fold. First, to the best of our knowledge, this is the first time that elemental discourse unit (EDU) has been introduced in multimodal summarization to guide the focus on critical, complete semantic information from both text and images for summary generation. Second, we propose a multimodal EDU selection module that accurately selects the critical EDU from the multimodal information. We conducted experiments on a widely used public dataset, and our model achieved state-of-the-art results on MMSS, demonstrating the effectiveness and superiority of our approach.