Multimodal Token Fusion and Optimization for 3D Human Mesh Reconstruction with Transformers
摘要
3D human recovery has attracted great attention and shown potential in games and movies. Confronting challenges related to occlusion and depth blurring in 3D human reconstruction, transformer encoder architectures have been used to make good progress in learning the connections between various parts of the human body. Nevertheless, for the input to the model, the differences between image tokens and vertex-joint tokens of different modalities are still limiting the reconstruction capability of the 3D human mesh. To overcome this limitation, we propose a module based on a multimodal cross-feature fusion mechanism directly fuses 2D images and 3D spatial coordinates to reconstruct a better human mesh. Our approach employs a large kernel attention strategy to improve the understanding of image features for spatial long-range relationships. We also design a token shift module for joints and vertices to learn interactions between vertices. Quantitative and qualitative experiments on large-scale human datasets such as 3DPW and Human3.6 show that our method achieves excellent reconstruction accuracy.