Unified Egocentric Dual-Hand Mesh Reconstruction and Action Recognition from Monocular RGB
摘要
Accurate 3D hand pose and shape estimation from monocular RGB images is crucial for egocentric vision applications such as augmented reality, human-computer interaction, and robotics. This task is very challenging due to frequent hand occlusions and cluttered, dynamic backgrounds typical in first-person views. To address these issues, we introduce a unified RGB-only framework that eliminates the need for depth sensors or multi-view setups. Our approach comprises three key components: (1) a transformer-based SHARP module that generates pseudo-depth maps for hand segmentation, (2) a 3D hand mesh reconstruction pipeline based on the MANO model with context-aware refinement for occlusion handling, and (3) a spatiotemporal transformer that integrates 3D hand poses with 2D object cues for robust action recognition. We evaluated the proposed model on the H2O dataset, achieving 96.53% accuracy for pose estimation and 94.97% top-1 accuracy in action recognition, which surpasses state-of-the-art methods. This framework provides a scalable solution for understanding egocentric interactions using only monocular RGB input, enabling practical deployment in wearable technology and real-time AR/VR systems.