Recognizing the interaction of people is a basic but challenging task in video understanding. In this paper, we propose a new task in this field, named Human-Human Interaction Recognition (HHIR). Our research focuses on recognizing interactions between two individuals in a scene, particularly those involving communication. It is different from the traditional action classification task. HHIR requires modeling complex and multimodal contextual cues across extended video scenes. To address this challenge, we propose a collaborative multimodal learning framework that integrates visual, textual, and audio modalities through an enhanced cross-attention mechanism. Our method leverages ViViT model for spatiotemporal visual feature extraction instead of traditional CNN models, BERT for dialogue understanding, and AST for audio feature extraction. Furthermore, we incorporate a collaborative knowledge integration strategy to enhance cross-modal interactions and apply class weighting to mitigate data imbalance issues. We build a new dataset, constructed from 12 diverse movies, capturing interactions between two individuals instead of isolated actions. Experimental results on this dataset demonstrate that our model achieves 78.54% Micro-F1 and 62.47% Macro-F1, outperforming state-of-the-art methods in multimodal human interaction recognition. This demonstrates the effectiveness of collaborative multimodal learning in capturing complex human interactions. The source code is available at https://github.com/hvngocthanh-CS/Human-Human-Interaction-Recognition-in-videos.git

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Collaborative Multimodal Learning for Human-Human Interaction Recognition in Videos

  • Vo Ngoc Thanh Huynh,
  • Thi Ngoc Ha Nguyen,
  • Tien-Dung Mai

摘要

Recognizing the interaction of people is a basic but challenging task in video understanding. In this paper, we propose a new task in this field, named Human-Human Interaction Recognition (HHIR). Our research focuses on recognizing interactions between two individuals in a scene, particularly those involving communication. It is different from the traditional action classification task. HHIR requires modeling complex and multimodal contextual cues across extended video scenes. To address this challenge, we propose a collaborative multimodal learning framework that integrates visual, textual, and audio modalities through an enhanced cross-attention mechanism. Our method leverages ViViT model for spatiotemporal visual feature extraction instead of traditional CNN models, BERT for dialogue understanding, and AST for audio feature extraction. Furthermore, we incorporate a collaborative knowledge integration strategy to enhance cross-modal interactions and apply class weighting to mitigate data imbalance issues. We build a new dataset, constructed from 12 diverse movies, capturing interactions between two individuals instead of isolated actions. Experimental results on this dataset demonstrate that our model achieves 78.54% Micro-F1 and 62.47% Macro-F1, outperforming state-of-the-art methods in multimodal human interaction recognition. This demonstrates the effectiveness of collaborative multimodal learning in capturing complex human interactions. The source code is available at https://github.com/hvngocthanh-CS/Human-Human-Interaction-Recognition-in-videos.git