Multimodal Prompts in AI-Assisted User Interface Design: How Many Words a Picture is Worth?
摘要
The progress in the development of generative models encouraged automated creation of user interfaces (UIs) based on textual prompts. Our study investigates the advantages of using multimodal prompts, which combine text and images, in AI-assisted UI design. In particular, we explored how more thorough textual descriptions compare to supplementation of shorter ones with style-defining pictures, wireframes, or screenshots of real interfaces. The 84 Uls that we generated with 3 different services have been evaluated by two professional UI/UX designers. Their positive feedback suggests that multimodal prompts can result in UI prototypes that have reasonably high Aesthetics, Level of Detail and Utility for the designers. Quantitative analysis demonstrated that the UIs generated with multimodal prompts received significantly higher mean evaluations: by 50.9% for Stable Diffusion 1.5, by 40.5% for KreaAI (which received the highest evaluations overall) and by 9.1% for UI/UX LoRA. The effect of the image type was somehow unexpected: better results (by 43.6% for KreaAI and by 32.4% for Stable Diffusion) were obtained with simpler wireframes (mock-ups) than with more detailed screenshots of real existing UIs. The results of our study might be of interest to HCI researchers and UI/UX practitioners who seek to incorporate AI technologies into UI design process.