Scholarlink

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published multimodal pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models, including both dense variants up to 30B and mixture-of-experts (MoE) variants up to 64B, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Brandon McKinzie,
  • Zhe Gan,
  • Jean-Philippe Fauconnier,
  • Sam Dodge,
  • Bowen Zhang,
  • Philipp Dufter,
  • Dhruti Shah,
  • Xianzhi Du,
  • Futang Peng,
  • Anton Belyi,
  • Haotian Zhang,
  • Karanjeet Singh,
  • Doug Kang,
  • Hongyu Hè,
  • Max Schwarzer,
  • Tom Gunter,
  • Xiang Kong,
  • Aonan Zhang,
  • Jianyu Wang,
  • Chong Wang,
  • Nan Du,
  • Tao Lei,
  • Sam Wiseman,
  • Mark Lee,
  • Zirui Wang,
  • Ruoming Pang,
  • Peter Grasch,
  • Alexander Toshev,
  • Yinfei Yang

摘要

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published multimodal pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models, including both dense variants up to 30B and mixture-of-experts (MoE) variants up to 64B, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.