Recent significant progress in the field of few-shot image generation has been achieved by fine-tuning pretrained text-to-image models, notably methods such as Dreambooth and Textual Inversion. To enhance the performance of existing methods, we advocate the explicit utilization of differences between various concepts to enable the generator to more effectively capture the characteristics of any given concept with limited samples. To this end, we introduce a contrastive generation approach that leverages these differences. Specifically, we expand the framework of Dreambooth by applying multimodal contrastive learning to optimize the feature distribution of conditions. Furthermore, we introduce another contrastive mechanism on the fused multimodal features between different concepts to further capture the characteristics of each new concept. Ultimately, our approach outperforms advanced few-shot generation models in capturing the characteristics of new concepts accurately. Additionally, it supports various applications and facilitates simultaneous learning of multiple new concepts without retraining for each.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Fine-Tuning Performance of Text-to-Image Diffusion Models for Few-Shot Image Generation Through Contrastive Learning

  • Yan-Lin Zhu,
  • Peipei Yang

摘要

Recent significant progress in the field of few-shot image generation has been achieved by fine-tuning pretrained text-to-image models, notably methods such as Dreambooth and Textual Inversion. To enhance the performance of existing methods, we advocate the explicit utilization of differences between various concepts to enable the generator to more effectively capture the characteristics of any given concept with limited samples. To this end, we introduce a contrastive generation approach that leverages these differences. Specifically, we expand the framework of Dreambooth by applying multimodal contrastive learning to optimize the feature distribution of conditions. Furthermore, we introduce another contrastive mechanism on the fused multimodal features between different concepts to further capture the characteristics of each new concept. Ultimately, our approach outperforms advanced few-shot generation models in capturing the characteristics of new concepts accurately. Additionally, it supports various applications and facilitates simultaneous learning of multiple new concepts without retraining for each.