Multimodal Large Language Models for Video Understanding
摘要
This chapter presents an innovative approach to video-language modeling, which enables video understanding across multiple modalities. The approach combines three key training strategies: masked video modeling, cross-modal contrastive learning, and next token prediction. The model architecture scales to 6B parameters and is trained with a progressive strategy. A distinguishing feature of this approach is its emphasis on spatiotemporal coherence, which is achieved through semantic video segmentation and multimodal caption generation that encompasses video, audio, and speech elements. This comprehensive framework demonstrates strong capabilities across diverse video understanding tasks, with particular strengths in video–text alignment and long-form video comprehension. The architecture enables sophisticated reasoning in video-centric dialog systems and shows robust performance in processing extended video sequences.