mmWave-based Hand Gesture Reconstruction
摘要
Hand gesture reconstruction plays a crucial role in enabling a range of interactive technologies such as human-computer interaction, sign language interpretation, virtual reality simulation, and more. Current methodologies predominantly rely on wearable devices like gloves or wristbands to capture hand gestures, which often involves high deployment costs and can result in a disruptive user experience. Alternatively, some techniques leverage vision-based systems, but these can suffer from challenges of varying lighting conditions and potential privacy concerns. In this chapter, we introduce mmHand, a novel system for 3D hand pose reconstruction based on mmWave radar signals. This approach enables the generation of 3D hand skeletons and the reconstruction of 3D hand meshes. First, mmHand uses mmWave signals to detect the hand and process the acquired mmWave data. Then, it applies a custom-designed attention-driven hourglass network, mmSpaceNet, to capture spatial features and employs LSTM networks to extract temporal characteristics. With the extracted features, mmHand then performs a regression to predict hand joint locations in 3D space, generating accurate 3D hand skeletons. Lastly, 3D hand meshes that provide a detailed and continuous representation of hand poses are reconstructed through a hand Model with Articulated and Non-rigid defOrmations (MANO). Extensive experimental evaluations show that mmHand achieves a mean per-joint position error of 18.3 mm and 95.1% accuracy, demonstrating its effectiveness in hand pose estimation.