Scholarlink

<p>In sports AI, human action recognition (HAR) faces a challenge between the expensive Transformer and the one-dimensional state space models (SSMs). Although Transformer has proven success on video tasks, its high computational cost scales quadratically. In contrast, conventional SSMs like Mamba possess linear complexity, but also underperform in the recognition. In this paper, we propose MMViT (Multi-scale Mamba Visual Transformer) with a hierarchical design for improved recognition and objective efficiency. We employ a heterogeneous “Attention-Mamba-Attention” (A-M-A) strategy. It first uses Multi-scale Pooling Attention (MPA) for efficient capture of local spatial feature. As computation-heavy stages come, it transitions to Mamba module with linear complexity to efficiently model long-range temporal context. Finally, attention is re-introduced at latter stages for semantic feature fusion. Also, we introduce a computation-downsampling decoupling (CDD) mechanism to preserve feature coverage during Mamba spatial scaling change. We have validated our approach on SpaceJam and Basketball-51 datasets. Experiments show that MMViT achieves superior performance over strong baselines with substantial margins. Ablation studies show the significance of A-M-A, MPA and CDD. MMViT achieves competitive accuracy among evaluated models and provides a favorable accuracy-efficiency trade-off for video action recognition task.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MMViT: Bridging Mamba and Attention for efficient video action recognition in sports

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月3日

查看全文
  • Jie Dai,
  • Ziyi Quan,
  • Yunxiang Zheng,
  • Zhengting Wang,
  • Peng Wang

摘要

In sports AI, human action recognition (HAR) faces a challenge between the expensive Transformer and the one-dimensional state space models (SSMs). Although Transformer has proven success on video tasks, its high computational cost scales quadratically. In contrast, conventional SSMs like Mamba possess linear complexity, but also underperform in the recognition. In this paper, we propose MMViT (Multi-scale Mamba Visual Transformer) with a hierarchical design for improved recognition and objective efficiency. We employ a heterogeneous “Attention-Mamba-Attention” (A-M-A) strategy. It first uses Multi-scale Pooling Attention (MPA) for efficient capture of local spatial feature. As computation-heavy stages come, it transitions to Mamba module with linear complexity to efficiently model long-range temporal context. Finally, attention is re-introduced at latter stages for semantic feature fusion. Also, we introduce a computation-downsampling decoupling (CDD) mechanism to preserve feature coverage during Mamba spatial scaling change. We have validated our approach on SpaceJam and Basketball-51 datasets. Experiments show that MMViT achieves superior performance over strong baselines with substantial margins. Ablation studies show the significance of A-M-A, MPA and CDD. MMViT achieves competitive accuracy among evaluated models and provides a favorable accuracy-efficiency trade-off for video action recognition task.