The rise of social media platforms has increased the demand for semantic-rich services, such as event and storyline attribution. However, most existing research focuses on clip-level event understanding, mainly through basic captioning tasks, without addressing the causal relationships between events across an entire movie. This presents a significant challenge, as even advanced multimodal large language models (MLLMs) struggle with extensive multimodal information due to limited context length. To tackle this, we propose a Two-Stage Prefix-Enhanced MLLM (TSPE) approach for event attribution, which connects events through their causal semantics in movie videos. In the local stage, we introduce an interaction-aware prefix to guide the model’s focus on relevant multimodal cues within a single clip, briefly summarizing each event. In the global stage, we enhance event connections using an inferential knowledge graph and design an event-aware prefix to focus on relevant events, not all preceding clips, leading to accurate event attribution. Extensive evaluations on two real-world datasets demonstrate that our framework outperforms state-of-the-art methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Generating Event-Oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM

  • Yuanjie Lyu,
  • Tong Xu,
  • Zihan Niu,
  • Bo Peng,
  • Jing Ke

摘要

The rise of social media platforms has increased the demand for semantic-rich services, such as event and storyline attribution. However, most existing research focuses on clip-level event understanding, mainly through basic captioning tasks, without addressing the causal relationships between events across an entire movie. This presents a significant challenge, as even advanced multimodal large language models (MLLMs) struggle with extensive multimodal information due to limited context length. To tackle this, we propose a Two-Stage Prefix-Enhanced MLLM (TSPE) approach for event attribution, which connects events through their causal semantics in movie videos. In the local stage, we introduce an interaction-aware prefix to guide the model’s focus on relevant multimodal cues within a single clip, briefly summarizing each event. In the global stage, we enhance event connections using an inferential knowledge graph and design an event-aware prefix to focus on relevant events, not all preceding clips, leading to accurate event attribution. Extensive evaluations on two real-world datasets demonstrate that our framework outperforms state-of-the-art methods.