VLM-Assisted Self-Healing GUI Testing with Hybrid CLIP SigLIP Matching
摘要
This paper investigates self-healing graphical user interface test automation assisted by modern vision-language models. It proposes a hybrid image-text matching framework that combines a contrastive image-text model with a sigmoid loss-based image-text model to locate and relink broken interface elements after software changes. The method first builds dense multimodal embeddings for widgets, screenshots, and textual descriptors collected from stable application versions. During regression testing, missing or relocated elements are identified by similarity search in this joint space, and candidate bindings are validated through structural constraints derived from the user interface hierarchy. To quantify the benefit of the hybrid design, we perform an ablation study that isolates the contributions of each component: visual encoding, textual encoding, multimodal fusion, and constraint-based filtering. Experiments on a benchmark of desktop and web applications show higher recovery accuracy and reduced manual repair time compared with traditional locator strategies and single-model baselines, demonstrating the potential of multimodal learning for robust interface testing.