Text2Omni: A Text-Only Training Strategy for MLLMs
摘要
Text2Omni is an innovative framework using only text modality to generate high-quality synthetic multimodal data, which aims to eliminate the need of large-scale annotated real multimodal data for training Multimodal Large Language Models (MLLMs). The framework leverages the geometric structure of multimodal contrastive representations to generate diverse, high-quality datasets that facilitate pretraining and instruction-tuning for multimodal models. The process involves a three-stage pipeline: (1) Diverse Caption Data Synthesis; (2) Instruction-Tuning Data Generation; and (3) Modality Representation Transfer. The resulting datasets, Text2Omni-1.8M for pretraining and Text2Omni-540K-Instruction for instruction-tuning, significantly reduce training costs while supporting the development of small- to medium-scale multimodal models. The paper also introduces a two-phase multimodal training paradigm to enhance multimodal understanding and reasoning capabilities of MLLMs efficiently. Experimental results across image-to-text and audio-to-text tasks demonstrate that the Text2Omni framework improves the performance of existing models on a variety of benchmarks, establishing its potential as an effective tool for advancing multimodal learning without requiring large-scale real-world data.