Graph Convolutional Network have emerged as a pivotal method in skeleton-based action recognition, demonstrating exceptional performance across multiple benchmarks. However, the inherent measurement errors in skeleton data—particularly the joint position estimation errors caused by occlusion—severely limit the recognition accuracy of existing models. To address this issue, this paper proposes a multimodal contrastive learning framework, PoseCLR, which effectively mitigates the impact of joint errors by leveraging the complementary characteristics of 2D and 3D skeleton data. Specifically, 3D skeleton data provides rich spatial information in three dimensions, while 2D skeleton data preserves more precise two-dimensional joint coordinates. The feature interaction between these two modalities achieves error compensation and information enhancement. Furthermore, this work supplements complete information on joint motion and bone motion, significantly improving the feature representation capability of individual modalities and thereby enhancing the performance of multi-stream score fusion. Experimental results demonstrate that the proposed method achieves breakthrough performance improvements on three mainstream datasets—NTU RGB+D, NTU RGB+D 120, and NW-UCLA—reaching current state-of-the-art methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PoseCLR: Bridging 2D and 3D Pose Representations via Contrastive Learning for Action Recognition

  • Jianlong Lu,
  • Weiyu Yu,
  • Jinquan Chen,
  • Dandan Qi,
  • Shuang Gong

摘要

Graph Convolutional Network have emerged as a pivotal method in skeleton-based action recognition, demonstrating exceptional performance across multiple benchmarks. However, the inherent measurement errors in skeleton data—particularly the joint position estimation errors caused by occlusion—severely limit the recognition accuracy of existing models. To address this issue, this paper proposes a multimodal contrastive learning framework, PoseCLR, which effectively mitigates the impact of joint errors by leveraging the complementary characteristics of 2D and 3D skeleton data. Specifically, 3D skeleton data provides rich spatial information in three dimensions, while 2D skeleton data preserves more precise two-dimensional joint coordinates. The feature interaction between these two modalities achieves error compensation and information enhancement. Furthermore, this work supplements complete information on joint motion and bone motion, significantly improving the feature representation capability of individual modalities and thereby enhancing the performance of multi-stream score fusion. Experimental results demonstrate that the proposed method achieves breakthrough performance improvements on three mainstream datasets—NTU RGB+D, NTU RGB+D 120, and NW-UCLA—reaching current state-of-the-art methods.