GenACT: Towards Unified Vision-Language-Action Co-training for Robot Manipulation
摘要
Vision-Language-Action models (VLAs) have emerged as a promising paradigm for building embodied agents, capable of acting in physical environments. Although often built from powerful Vision-Language models (VLMs), most existing VLAs cannot fully exploit the capacity of large foundation models, since they rely heavily on limited-scale action-only supervision. This narrow paradigm also limits their capabilities to leverage the internet-scale multimodal knowledge, thereby constraining their generalization capabilities. In this paper, we present GenACT (generative action transformer), a unified robot foundation model that moves beyond purely action-centric supervision and unifies modalities by tokenizing visual, language, and action into a shared discrete space. GenACT aims at unified vision-language-action co-learning which consists of two stages: (1) multimodal generative pretraining with a large language model and (2) multimodal action co-training. We adopt next token prediction as the training objective, where the model learns to predict future tokens across all modalities in a self-supervised manner, for a unified and flexible training paradigm. Hence, given language instruction and visual observation, the GenACT autoregressively generates action predictions. The experiments suggest that our training paradigm significantly benefits robot manipulation, and consequently GenACT achieves state-of-the-art (SOTA) manipulation performance on both simulation and real robot benchmarks. We also conduct thorough experiments to identify the key design choices.