From Vision to Grasp: A Multi-device Hand Tracking System for Robotic Teleoperation in Daily Activities
摘要
As global populations age, enabling older adults to live independently becomes increasingly important. This study presents a non-invasive, multi-device dexterous hand teleoperation system designed to enhance domestic service robots’ ability to perform assistive tasks remotely. By integrating multiple visual-based sensors—specifically cameras and Leap Motion Controllers—the system effectively recognizes two common hand grasping gestures, allowing real-time, accurate control of a robotic manipulator. Unlike prior approaches that rely on single-sensor input or wearable devices, the proposed system leverages sensor redundancy to improve gesture recognition and user comfort. Performance was validated through user studies involving two daily tasks, demonstrating a high success rate (94.17%) and positive user experience, with training effects observed in task efficiency and perceived workload. Notably, the system showed consistent usability and reliability across tasks, with NASA-TLX analysis highlighting decreased cognitive and physical demand over time. Future work aims to enhance grasp recognition with additional hand features, compare with wearable alternatives, and extend control to full robotic arm kinematics for more complex tasks. This work contributes toward scalable, intuitive teleoperation technologies that support aging in place through robotic assistance.