<p>Diffusion models have made remarkable progress in generating high-quality images based on user input prompts. However, most open-source models are designed to support only English text prompts, creating challenges for users of non-English languages. Commonly used translation methods fail to address cultural nuances in image generation, and training models from scratch on language-specific data is prohibitively costly. This study introduces a lightweight solution for training low-resource text encoders, using a cross-lingual teacher-student learning approach to guide the image generation process in Stable Diffusion effectively. Extensive experiments on both general and culturally relevant benchmarks were conducted to assess image quality, alignment with human preferences, and visual appeal. In the Thai cultural benchmark, our method outperforms GlueGen as reported by Qin et al (Proceedings of the IEEE/CVF International Conference on Computer Vision, 23085–23096, 2023.), the current state-of-the-art PEFT technique, achieving 100% winning rate on CLIPScore, ImageReward, and PickScore, with 81% on VQAScore and 85% on Aesthetic Score. Importantly, the integration of enriched synthetic data significantly improves both performance and efficiency in cross-lingual learning across all benchmarks. These advancements offer a pathway for broader adoption of text-to-image models for low-resource languages.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Optimizing low-resource language encoders for text-to-image generation: a case study on Thai

  • Thitirat Siriborvornratanakul,
  • Songpol Bunyang

摘要

Diffusion models have made remarkable progress in generating high-quality images based on user input prompts. However, most open-source models are designed to support only English text prompts, creating challenges for users of non-English languages. Commonly used translation methods fail to address cultural nuances in image generation, and training models from scratch on language-specific data is prohibitively costly. This study introduces a lightweight solution for training low-resource text encoders, using a cross-lingual teacher-student learning approach to guide the image generation process in Stable Diffusion effectively. Extensive experiments on both general and culturally relevant benchmarks were conducted to assess image quality, alignment with human preferences, and visual appeal. In the Thai cultural benchmark, our method outperforms GlueGen as reported by Qin et al (Proceedings of the IEEE/CVF International Conference on Computer Vision, 23085–23096, 2023.), the current state-of-the-art PEFT technique, achieving 100% winning rate on CLIPScore, ImageReward, and PickScore, with 81% on VQAScore and 85% on Aesthetic Score. Importantly, the integration of enriched synthetic data significantly improves both performance and efficiency in cross-lingual learning across all benchmarks. These advancements offer a pathway for broader adoption of text-to-image models for low-resource languages.