Scholarlink

<p>Multimodal video summarization is crucial for reducing redundancy and highlighting key moments in long or complex videos. However, current methods face challenges in integrating multimodal information deeply and handling their semantic relations. To address these issues, we propose GraphVSum, a novel framework that converts multimodal video summarization into a heterogeneous graph processing task. It constructs a heterogeneous graph of video frames and associated sentences, and optimizes their connections through intra-modal and inter-modal constraints. In GraphVSum, a dual fusion technique is proposed for multimodal embedding, combining shallow fusion with global attention and deep fusion via subgraph propagation. For generating accurate and contextually rich summaries, we propose a novel multimodal joint learning in GraphVSum, which includes a classification loss, an inter-modal coherence loss and an intra-modal diversity loss. Extensive experiments on the BLiSS, PlotSnap, Daily Mail and TVSum datasets demonstrate the effectiveness of GraphVSum in several metrics. The code of GraphVSum is publicly available at <a href="https://github.com/eMoLii/GraphVSum">https://github.com/eMoLii/GraphVSum</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GraphVSum:graph guided multimodal video summarization

  • 期刊论文

  • 发表日期：

    2025年12月6日

查看全文
  • Zhengqi Zhao,
  • Cong Bai,
  • Pengyi Hao

摘要

Multimodal video summarization is crucial for reducing redundancy and highlighting key moments in long or complex videos. However, current methods face challenges in integrating multimodal information deeply and handling their semantic relations. To address these issues, we propose GraphVSum, a novel framework that converts multimodal video summarization into a heterogeneous graph processing task. It constructs a heterogeneous graph of video frames and associated sentences, and optimizes their connections through intra-modal and inter-modal constraints. In GraphVSum, a dual fusion technique is proposed for multimodal embedding, combining shallow fusion with global attention and deep fusion via subgraph propagation. For generating accurate and contextually rich summaries, we propose a novel multimodal joint learning in GraphVSum, which includes a classification loss, an inter-modal coherence loss and an intra-modal diversity loss. Extensive experiments on the BLiSS, PlotSnap, Daily Mail and TVSum datasets demonstrate the effectiveness of GraphVSum in several metrics. The code of GraphVSum is publicly available at https://github.com/eMoLii/GraphVSum.