Diffusion models have emerged as a powerful tool for generative tasks, showing impressive capabilities in various domains. This chapter explores the potential of the diffusion U-Net architecture, highlighting how it can serve as an effective and straightforward approach to enhancing generation quality in generative models. Through an analysis of the U-Net structure, it becomes clear that the backbone is primarily responsible for denoising, while the skip connections play a crucial role in introducing high-frequency details into the decoderDecoder. However, these skip connections can sometimes overshadow the backbone’s key functions. By reconsidering the interaction between the skip connections and the backbone, it is possible to significantly improve generation quality without the need for additional training or fine-tuning. This approach, referred to as FreeU, involves re-weighting the contributions of these two components to leverage their strengths. Promising results on tasks such as image and video generation demonstrate how FreeU can be easily integrated into existing diffusion models like Stable Diffusion, DreamBooth, and ControlNet, requiring only a few adjustments during inference. The simplicity of the method—adjusting two scaling factors—makes it a powerful tool for improving generation quality with minimal changes to the underlying model.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Boosting Diffusion U-Net with Free Lunch for Text-to-Image and Text-to-Video Generation

  • Chenyang Si,
  • Ziqi Huang,
  • Yuming Jiang,
  • Ziwei Liu

摘要

Diffusion models have emerged as a powerful tool for generative tasks, showing impressive capabilities in various domains. This chapter explores the potential of the diffusion U-Net architecture, highlighting how it can serve as an effective and straightforward approach to enhancing generation quality in generative models. Through an analysis of the U-Net structure, it becomes clear that the backbone is primarily responsible for denoising, while the skip connections play a crucial role in introducing high-frequency details into the decoderDecoder. However, these skip connections can sometimes overshadow the backbone’s key functions. By reconsidering the interaction between the skip connections and the backbone, it is possible to significantly improve generation quality without the need for additional training or fine-tuning. This approach, referred to as FreeU, involves re-weighting the contributions of these two components to leverage their strengths. Promising results on tasks such as image and video generation demonstrate how FreeU can be easily integrated into existing diffusion models like Stable Diffusion, DreamBooth, and ControlNet, requiring only a few adjustments during inference. The simplicity of the method—adjusting two scaling factors—makes it a powerful tool for improving generation quality with minimal changes to the underlying model.