Automatic Code Generation from GUI Screenshots with Vision-Language Models
摘要
There are two challenges in the GUI code generation tasks: insufficient image feature extraction capability and low accuracy of code generation. Existing methods can handle simple GUI datasets well but have difficulty with complex GUI datasets. We propose a method for automatic code generation from GUI screenshots with vision-language models. Our method uses a multimodal information fusion module to fuse feature information from images and code efficiently. To improve the accuracy and feature extraction capability of GUI code generation, we use a combination of automated metrics to evaluate the performance of our method. The multiple experimental results demonstrate that our method achieves better performance among existing methods on both existing public and newly constructed datasets in the GUI code generation field. In particular, the BLUE4 score has been improved by 27% on the newly constructed dataset. The vision-language model architecture in our method can meet the diversity of code generation tasks and provide a helpful solution for GUI development in various domains.