<p>The field of action recognition has primarily focused on identifying complex actions in high-quality videos, yet recognizing low-resolution movements remains a critical challenge with practical applications in real-world surveillance scenarios. This paper proposes a dual-branch vision transformer to address the spatiotemporal feature learning challenges in low-resolution action recognition. Our architecture employs distinct processing pathways: the primary branch captures spatial patterns using reduced frame rates, whereas the auxiliary branch operates at higher temporal resolution with spatial downsampling. Continuous feature interaction between branches is facilitated through cross-attention modules, enabling complementary spatiotemporal learning. This architectural separation allows targeted feature extraction: spatial analysis benefits from temporal context refinement while motion patterns gain enhanced spatial grounding. To validate our approach, we contributed a novel classroom behavior dataset containing 7,722 annotated videos (6,776 training / 1,546 tests) across 11 student action categories, captured through university surveillance systems. Experimental results demonstrate state-of-the-art performance with 75.20% F1-score on TinyVIRAT-v2 and 83.41% top-1 accuracy on our classroom dataset, outperforming existing baselines with only 13.68% additional computational overhead compared to single-branch transformers.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-branch vision transformer for low-resolution action recognition

  • Ruixin Chen,
  • Chenqiang Gao,
  • Zhuolin Tan,
  • Fangxin Liu,
  • Jiayi Yu,
  • Xinlin Li

摘要

The field of action recognition has primarily focused on identifying complex actions in high-quality videos, yet recognizing low-resolution movements remains a critical challenge with practical applications in real-world surveillance scenarios. This paper proposes a dual-branch vision transformer to address the spatiotemporal feature learning challenges in low-resolution action recognition. Our architecture employs distinct processing pathways: the primary branch captures spatial patterns using reduced frame rates, whereas the auxiliary branch operates at higher temporal resolution with spatial downsampling. Continuous feature interaction between branches is facilitated through cross-attention modules, enabling complementary spatiotemporal learning. This architectural separation allows targeted feature extraction: spatial analysis benefits from temporal context refinement while motion patterns gain enhanced spatial grounding. To validate our approach, we contributed a novel classroom behavior dataset containing 7,722 annotated videos (6,776 training / 1,546 tests) across 11 student action categories, captured through university surveillance systems. Experimental results demonstrate state-of-the-art performance with 75.20% F1-score on TinyVIRAT-v2 and 83.41% top-1 accuracy on our classroom dataset, outperforming existing baselines with only 13.68% additional computational overhead compared to single-branch transformers.