Accurate 3D hand pose and shape estimation from monocular RGB images is crucial for egocentric vision applications such as augmented reality, human-computer interaction, and robotics. This task is very challenging due to frequent hand occlusions and cluttered, dynamic backgrounds typical in first-person views. To address these issues, we introduce a unified RGB-only framework that eliminates the need for depth sensors or multi-view setups. Our approach comprises three key components: (1) a transformer-based SHARP module that generates pseudo-depth maps for hand segmentation, (2) a 3D hand mesh reconstruction pipeline based on the MANO model with context-aware refinement for occlusion handling, and (3) a spatiotemporal transformer that integrates 3D hand poses with 2D object cues for robust action recognition. We evaluated the proposed model on the H2O dataset, achieving 96.53% accuracy for pose estimation and 94.97% top-1 accuracy in action recognition, which surpasses state-of-the-art methods. This framework provides a scalable solution for understanding egocentric interactions using only monocular RGB input, enabling practical deployment in wearable technology and real-time AR/VR systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unified Egocentric Dual-Hand Mesh Reconstruction and Action Recognition from Monocular RGB

  • Kamal,
  • Abhisekh Roy,
  • Liza Kalita,
  • Debanga Raj Neog,
  • M. K. Bhuyan

摘要

Accurate 3D hand pose and shape estimation from monocular RGB images is crucial for egocentric vision applications such as augmented reality, human-computer interaction, and robotics. This task is very challenging due to frequent hand occlusions and cluttered, dynamic backgrounds typical in first-person views. To address these issues, we introduce a unified RGB-only framework that eliminates the need for depth sensors or multi-view setups. Our approach comprises three key components: (1) a transformer-based SHARP module that generates pseudo-depth maps for hand segmentation, (2) a 3D hand mesh reconstruction pipeline based on the MANO model with context-aware refinement for occlusion handling, and (3) a spatiotemporal transformer that integrates 3D hand poses with 2D object cues for robust action recognition. We evaluated the proposed model on the H2O dataset, achieving 96.53% accuracy for pose estimation and 94.97% top-1 accuracy in action recognition, which surpasses state-of-the-art methods. This framework provides a scalable solution for understanding egocentric interactions using only monocular RGB input, enabling practical deployment in wearable technology and real-time AR/VR systems.