The rise of multimodal large language models (LLMs) has led to a surge of GUI navigation agents, which aim to automate GUI operations following human instructions. Implementing such an agent implies being able to reason action intents at each step from instructions and to ground them on the GUI environment. Previous work entangle the reasoning and grounding processes by direct coordinate prediction, ignoring their distinct requirements of information at different abstraction levels. In this paper, we propose Reasoning and Grounding GUI Automation agent (ReGA), an multimodal LLM-based agent that decouples these two processes. We incorporate additional tokens into the existing vocabulary of the LLM as the representation of intended location, whose hidden embeddings are further decoded into coordinates. In this way, we transfer the grounding function from language model to the location decoder. Upon this design, we further enhance the grounding capability of location decoder by fusing low-level vision features, and improve the reasoning performance by high-level task-generic plans named Generic Interaction Plans. Experimental results shows that ReGA achieves state-of-the-art performance on representative GUI navigation benchmarks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ReGA: Reasoning and Grounding Decoupled GUI Navigation Agents

  • Feiyue Ni,
  • Yanchu Guan,
  • Yuchong Sun,
  • Dong Wang,
  • Chenyi Zhuang,
  • Jinjie Gu,
  • Ruihua Song

摘要

The rise of multimodal large language models (LLMs) has led to a surge of GUI navigation agents, which aim to automate GUI operations following human instructions. Implementing such an agent implies being able to reason action intents at each step from instructions and to ground them on the GUI environment. Previous work entangle the reasoning and grounding processes by direct coordinate prediction, ignoring their distinct requirements of information at different abstraction levels. In this paper, we propose Reasoning and Grounding GUI Automation agent (ReGA), an multimodal LLM-based agent that decouples these two processes. We incorporate additional tokens into the existing vocabulary of the LLM as the representation of intended location, whose hidden embeddings are further decoded into coordinates. In this way, we transfer the grounding function from language model to the location decoder. Upon this design, we further enhance the grounding capability of location decoder by fusing low-level vision features, and improve the reasoning performance by high-level task-generic plans named Generic Interaction Plans. Experimental results shows that ReGA achieves state-of-the-art performance on representative GUI navigation benchmarks.