ViT-DtC: vision transformer-based design-to-code framework for code generation from generated UI designs and hand-drawn sketches
摘要
Code generation in software development from various types of user interface (UI) design images can significantly reduce the manual effort required, accelerate development timelines, and facilitate collaboration between designers and developers. Previous studies have restricted the utilized types of UI designs to either generated UI designs or hand-drawn sketches separately, lacking the automatic detection capability for the design type. Moreover, studies frequently necessitated complex datasets with bounding box annotations and sophisticated computer vision preprocessing steps, especially within the context of hand-drawn sketches. In this paper, we introduce the novel comprehensive Vision Transformer-based Design-to-Code (ViT-DtC) framework, which tackles image classification and code generation from both generated UI designs and hand-drawn sketches by harnessing the capabilities of Vision Transformers. The proposed ViT-DtC undergoes fine-tuning to classify UI designs into specific categories, including web, iOS, Android, and hand-drawn sketches. Subsequently, the design is directed to the appropriate code generator to generate domain-specific language (DSL) tokens for UI elements based on the identified design type. The experimental results exhibited an exceptional proficiency in accurately classifying all designs. Employing a greedy search strategy, the proposed ViT-DtC framework achieved an average accuracy of 97.28% in generating UI elements on the modified web dataset (without capturing their state or color information). In iOS and Android designs, an average accuracy of 82.4% and 81.1% was achieved, respectively. Remarkably, when extended to handle hand-drawn sketches, an average accuracy of 84.8% was maintained.