One of the most popular tools for generating specific images is diffusion models, which can model the input prompt into a realistic picture. They have been trained on huge datasets, so they can create photographs of any topic. However, they have become too general networks that have problems generating specific motifs or styles, and their retraining is too complex computationally. Therefore, a technique has been proposed: low-rank adaptation (LoRA), which can retrain a diffusion model of specific styles without having to train the entire model from scratch, but by creating smaller weight matrices that will allow relearning specific information. However, in practice, such a solution requires knowledge of appropriate trigger words that allow the activation of specific adapters. Therefore, we propose a lightweight system that allows for generating images without the need to use predefined style tokens or user intervention. Our approach focuses on the analysis of the user prompt by embedding CLIP and comparison of text n-grams with previously prepared style representations. Our experiments confirm that our method effectively identifies styles, which makes it a practical tool for intelligent image editing.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Injection of LoRA Adapters via Prompt Semantics for Modular Stylization in Diffusion Models

  • Alicja Polowczyk,
  • Agnieszka Polowczyk

摘要

One of the most popular tools for generating specific images is diffusion models, which can model the input prompt into a realistic picture. They have been trained on huge datasets, so they can create photographs of any topic. However, they have become too general networks that have problems generating specific motifs or styles, and their retraining is too complex computationally. Therefore, a technique has been proposed: low-rank adaptation (LoRA), which can retrain a diffusion model of specific styles without having to train the entire model from scratch, but by creating smaller weight matrices that will allow relearning specific information. However, in practice, such a solution requires knowledge of appropriate trigger words that allow the activation of specific adapters. Therefore, we propose a lightweight system that allows for generating images without the need to use predefined style tokens or user intervention. Our approach focuses on the analysis of the user prompt by embedding CLIP and comparison of text n-grams with previously prepared style representations. Our experiments confirm that our method effectively identifies styles, which makes it a practical tool for intelligent image editing.