Dual-Image-Guided Grounding of User Interfaces with MLLMs
摘要
Recent innovations in large-scale multimodal language models (MLLM) have significantly improved the anchoring of graphical interfaces, a process that involves linking natural language instructions to specific interface elements. Performance tests such as ScreenSpot Pro have demonstrated the level of difficulty of anchoring in high-resolution professional scenarios. In such situations, small elements and complex layouts make it difficult for the model to function properly. However, currently, only the evaluation of a single image is taken into account, and the ability of MLLM to reason from multiple visual inputs is ignored. In this work, we propose the dual image prompting for GUI grounding approach. The model receives a template image that designates the target area and a full screen image that represents the candidate region. It must then find the template in the screen. To do this, it combines conventional template matching with semantic reasoning and multi-image alignment.