Diffusion-based models have demonstrated remarkable capabilities in generating high-quality images from text and images. Applying these techniques to generate procedural tutorial sequences for guiding human activities in areas such as crafting, painting, and modeling is both meaningful and challenging. Datasets for procedural sequences across different domains are scarce, and current methods that generalize procedural knowledge from existing domains to others via one-shot learning often lack robustness and precision. In this paper, we introduce StrategyAdapter, a highly versatile and parameter-efficient approach that injects strategy information from procedural sequences across various domains into a pre-trained Diffusion Transformer. This enables one-shot generation of procedurally consistent and visually coherent sequences in unseen domains for both text-to-sequence and image-to-sequence tasks. Extensive experiments demonstrate that StrategyAdapter outperforms existing methods in procedural generation tasks within unseen domains. Additionally, we have constructed a high-quality procedural dataset comprising over 12,000 sequences across 12 categories, advancing research in procedural generation. Code is available at https://github.com/LUYserena/StrategyAdapter , as the anonymous repository was only used for double-blind review.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

StrategyAdapter: One-Shot Learning for Unseen-Domain Procedural Sequence Generation

  • Yingying Sun,
  • Zhiguang Chen,
  • Nong Xiao

摘要

Diffusion-based models have demonstrated remarkable capabilities in generating high-quality images from text and images. Applying these techniques to generate procedural tutorial sequences for guiding human activities in areas such as crafting, painting, and modeling is both meaningful and challenging. Datasets for procedural sequences across different domains are scarce, and current methods that generalize procedural knowledge from existing domains to others via one-shot learning often lack robustness and precision. In this paper, we introduce StrategyAdapter, a highly versatile and parameter-efficient approach that injects strategy information from procedural sequences across various domains into a pre-trained Diffusion Transformer. This enables one-shot generation of procedurally consistent and visually coherent sequences in unseen domains for both text-to-sequence and image-to-sequence tasks. Extensive experiments demonstrate that StrategyAdapter outperforms existing methods in procedural generation tasks within unseen domains. Additionally, we have constructed a high-quality procedural dataset comprising over 12,000 sequences across 12 categories, advancing research in procedural generation. Code is available at https://github.com/LUYserena/StrategyAdapter , as the anonymous repository was only used for double-blind review.