Scholarlink

<p>Given the remarkable achievements in image generation using diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational cost, which increases quadratically with sequence length. This limitation poses significant challenges when generating longer video sequences using diffusion models. To overcome these challenges, we propose to leverage state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently garnered attention as promising alternatives owing to their linear-time memory and time consumption relative to the sequence length. Employing SSMs to capture temporal dependencies in video generation enables significantly higher generative performance at the same computational cost (e.g., memory usage, inference time) compared to attention-based methods, particularly for long-term sequences. For various model sizes, we comprehensively evaluated multiple long-term video datasets: MineRL Navigate, GQN-Mazes, and CARLA-Town01. For 256-frame video sequences, SSM-based models incur lower computational cost to achieve the same Fréchet Video Distance as attention-based models. Furthermore, the ablation study shows that when using SSMs for temporal modeling, incorporating bidirectionality and selective scans enhances video generation performance. Our code is available at <a href="https://anonymous.4open.science/r/SSM-Meets-Video-Diffusion-Models-067D/README.md">https://anonymous.4open.science/r/SSM-Meets-Video-Diffusion-Models-067D/README.md</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月27日

查看全文
  • Yuta Oshima,
  • Shohei Taniguchi,
  • Masahiro Suzuki,
  • Yutaka Matsuo

摘要

Given the remarkable achievements in image generation using diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational cost, which increases quadratically with sequence length. This limitation poses significant challenges when generating longer video sequences using diffusion models. To overcome these challenges, we propose to leverage state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently garnered attention as promising alternatives owing to their linear-time memory and time consumption relative to the sequence length. Employing SSMs to capture temporal dependencies in video generation enables significantly higher generative performance at the same computational cost (e.g., memory usage, inference time) compared to attention-based methods, particularly for long-term sequences. For various model sizes, we comprehensively evaluated multiple long-term video datasets: MineRL Navigate, GQN-Mazes, and CARLA-Town01. For 256-frame video sequences, SSM-based models incur lower computational cost to achieve the same Fréchet Video Distance as attention-based models. Furthermore, the ablation study shows that when using SSMs for temporal modeling, incorporating bidirectionality and selective scans enhances video generation performance. Our code is available at https://anonymous.4open.science/r/SSM-Meets-Video-Diffusion-Models-067D/README.md.