Scholarlink

Multi-modal action recognition, which leverages spatial and temporal information from various data streams, has become crucial in computer vision. This paper traces the evolution of our research from a model employing Dual Attention Convolutional Neural Networks (DA-CNNs) with Bidirectional Gated Recurrent Units (Bi-GRUs) to a sophisticated hybrid architecture integrating EfficientNet for feature extraction, custom CNN layers for feature refinement, and Vision Transformers (ViTs) for temporal modeling. This work was developed for the ICPR 2024 Multi-Modal Visual Pattern Recognition Challenge (Track 3: Multi-Modal Action Recognition). The dataset comprises 2,500 multi-modal videos (2,000 for training and 500 for testing) across 20 action classes, each including synchronized RGB, Infrared (IR), and Depth streams with varying resolutions and durations. Our final model processes these diverse data streams to achieve robust action recognition. We present a thorough analysis of both our initial and advanced architectures, including detailed mathematical formulations and experimental results. Notably, our approach attained an accuracy of 87.8%, surpassing the baseline model’s performance of 86.43%. This research contributes significantly to the ongoing development of multi-modal action recognition techniques, with potential applications in various domains like surveillance, human-computer interaction, and autonomous systems.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evolution of Hybrid Multi-modal Action Recognition: From DA-CNN+Bi-GRU to EfficientNet-CNN-ViT

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Adars Thazhayamkotath Mana Subramanian,
  • Anagha S. Menon,
  • Vaishnav Reghunath,
  • Jyothisha J. Nair,
  • Akhil Kalathiparambil Asokakumar,
  • Nandagopal Jayagopal

摘要

Multi-modal action recognition, which leverages spatial and temporal information from various data streams, has become crucial in computer vision. This paper traces the evolution of our research from a model employing Dual Attention Convolutional Neural Networks (DA-CNNs) with Bidirectional Gated Recurrent Units (Bi-GRUs) to a sophisticated hybrid architecture integrating EfficientNet for feature extraction, custom CNN layers for feature refinement, and Vision Transformers (ViTs) for temporal modeling. This work was developed for the ICPR 2024 Multi-Modal Visual Pattern Recognition Challenge (Track 3: Multi-Modal Action Recognition). The dataset comprises 2,500 multi-modal videos (2,000 for training and 500 for testing) across 20 action classes, each including synchronized RGB, Infrared (IR), and Depth streams with varying resolutions and durations. Our final model processes these diverse data streams to achieve robust action recognition. We present a thorough analysis of both our initial and advanced architectures, including detailed mathematical formulations and experimental results. Notably, our approach attained an accuracy of 87.8%, surpassing the baseline model’s performance of 86.43%. This research contributes significantly to the ongoing development of multi-modal action recognition techniques, with potential applications in various domains like surveillance, human-computer interaction, and autonomous systems.