SegCL: Segmented Reasoning with Global Visual-Audio Knowledge for Complex Long Video Understanding
摘要
Multimodal large language models have been widely applied in the field of video understanding, but mainly for simple short videos. Due to the length and complex plots, understanding complex long videos remains a significant challenge. To handle longer videos, previous research have tried optimizing sampling strategies and compressing tokens to address the reasoning challenges posed by large numbers of tokens. However, these methods only relatively increase the length of videos that can be processed, and the loss of visual information becomes more severe as video length increases. To deal with more complex plots, there has also been a growing focus on audio information, but primarily through extracting transcription data from subtitles. To better understand complex long video, we propose SegCL, a zero-shot reasoning method without additional information. It can perform local reasoning with global audio-character information, theoretically enables reasoning infinitely long videos for its unique design. SegCL extracts global character information by extracting facial features and clustering, extracts audio information by transcription and speaker diarization from the video, and then matches dialogue and character with a novel algorithm as global audio-character information. Then videos are segmented into slices by scene, MLLM reasons each slice with its global information. Ultimately LLM extract character knowledge graph and based on this summarize the detailed content of complex long videos. To evaluate the performance of SegCL, we also introduce an evaluation set for complex long video. Under the dual evaluation mechanism of gpt_4o and humans, SegCL demonstrates comprehensive performance improvements over its base model in understanding complex long videos.