Scholarlink

<p>Diffusion models have achieved unprecedented fidelity and diversity in synthesizing images, videos, and 3D assets. However, <i>subject mixing</i> remains an unresolved issue in diffusion-based image synthesis, particularly when synthesizing multiple subjects from similar categories (e.g., <i>“a shiba and a husky”</i>) or the same category (e.g., <i>“five backpacks”</i>). We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared with previous methods that rely solely on self-attention or cross-attention, and the prompt-engineering baseline, our guidance is more effective at eliminating subject mixing. It addresses subject mixing across all relevant patches rather than only the most discriminative one (e.g., a bird’s beak). For each subject, we aggregate the self-attention maps of patches with higher cross-attention values. Thus, the resulting aggregated map captures the region attended to by the entire subject. We also identify the issue in synthesizing multiple objects of the same category and propose an instance segmentation method that leverages the attention heads of diffusion models, formulating our guidance between segments. Our training-free method improves the performance of both U-Net-based and Transformer-based diffusion models, including the Stable Diffusion series and FLUX series. We also adapt self-cross guidance as an effective reward for RL-based post-training and show improved subject diversity with no computational overhead at inference time. Such a reward is computed internally during generation and doesn’t rely on any external model. We also release a challenging benchmark, the Similar Subjects Dataset (SSD), and employ GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results, including a user study, demonstrate the effectiveness of our Self-Cross Diffusion Guidance. The code is available at <a href="https://github.com/mengtang-lab/selfcross-guidance">https://github.com/mengtang-lab/selfcross-guidance</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Diversifying Similar Subjects for Text-to-Image Synthesis with Self-Cross Diffusion Guidance and Reward

  • 期刊论文

  • 发表日期：

    2026年7月23日

查看全文
  • Weimin Qiu,
  • Jieke Wang,
  • Zhining Gu,
  • Vincent Tao Hu,
  • Meng Tang

摘要

Diffusion models have achieved unprecedented fidelity and diversity in synthesizing images, videos, and 3D assets. However, subject mixing remains an unresolved issue in diffusion-based image synthesis, particularly when synthesizing multiple subjects from similar categories (e.g., “a shiba and a husky”) or the same category (e.g., “five backpacks”). We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared with previous methods that rely solely on self-attention or cross-attention, and the prompt-engineering baseline, our guidance is more effective at eliminating subject mixing. It addresses subject mixing across all relevant patches rather than only the most discriminative one (e.g., a bird’s beak). For each subject, we aggregate the self-attention maps of patches with higher cross-attention values. Thus, the resulting aggregated map captures the region attended to by the entire subject. We also identify the issue in synthesizing multiple objects of the same category and propose an instance segmentation method that leverages the attention heads of diffusion models, formulating our guidance between segments. Our training-free method improves the performance of both U-Net-based and Transformer-based diffusion models, including the Stable Diffusion series and FLUX series. We also adapt self-cross guidance as an effective reward for RL-based post-training and show improved subject diversity with no computational overhead at inference time. Such a reward is computed internally during generation and doesn’t rely on any external model. We also release a challenging benchmark, the Similar Subjects Dataset (SSD), and employ GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results, including a user study, demonstrate the effectiveness of our Self-Cross Diffusion Guidance. The code is available at https://github.com/mengtang-lab/selfcross-guidance.