Reconstructing 3D hand mesh from a monocular RGB image is pivotal for human-computer interaction. However, this task becomes highly challenging due to severe occlusions during hand-object interactions. Existing methods suffer from fixed convolutional kernels of CNNs and unreliable similarity computations in Transformers. Consequently, they fail to solve dynamic occlusions and background interference, leading to pose inaccuracies and mesh distortions. To overcome these challenges, we propose HandDSA, an occlusion-robust network that dynamically exploits spatial-aware information to enhance reconstruction quality. First, we propose the Spatial Attention-Guided Dynamic Convolution (SAD-Conv) that dynamically generates region-specific kernels by analyzing spatial patterns. Specifically, smaller receptive fields are employed in visible hand regions to obtain detailed local features while larger receptive fields are utilized in occluded interaction regions to capture global contextual semantics. Then, we design the Gated Attention Filtering (GAF) that combines channel-spatial attention with gated mechanism to adaptively suppress irrelevant interference from background regions. Extensive experiments demonstrate the efficacy of our approach across multiple evaluation metrics on the HO3Dv2 and DexYCB datasets.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Spatial-Aware Network for Occlusion Robust 3D Hand Mesh Reconstruction from RGB Images

  • Yibo Bai,
  • Xiaoqing Yin,
  • Zhengbin Pang,
  • Jinsheng Deng

摘要

Reconstructing 3D hand mesh from a monocular RGB image is pivotal for human-computer interaction. However, this task becomes highly challenging due to severe occlusions during hand-object interactions. Existing methods suffer from fixed convolutional kernels of CNNs and unreliable similarity computations in Transformers. Consequently, they fail to solve dynamic occlusions and background interference, leading to pose inaccuracies and mesh distortions. To overcome these challenges, we propose HandDSA, an occlusion-robust network that dynamically exploits spatial-aware information to enhance reconstruction quality. First, we propose the Spatial Attention-Guided Dynamic Convolution (SAD-Conv) that dynamically generates region-specific kernels by analyzing spatial patterns. Specifically, smaller receptive fields are employed in visible hand regions to obtain detailed local features while larger receptive fields are utilized in occluded interaction regions to capture global contextual semantics. Then, we design the Gated Attention Filtering (GAF) that combines channel-spatial attention with gated mechanism to adaptively suppress irrelevant interference from background regions. Extensive experiments demonstrate the efficacy of our approach across multiple evaluation metrics on the HO3Dv2 and DexYCB datasets.