Touchless interaction has emerged as a pivotal feature for seamless user experience in the field of wearable technology. This paper presents a low-power time-of-flight (ToF) based gesture recognition system tailored for touchless wearables. We leverage the precision of ToF sensors and the advances in edge AI to detect and interpret a variety of finger gestures, enabling a more intuitive and responsive user interface. Both Temporal Convolutional Network (TCN) and Long Short-Term Memory (LSTM) architectures are evaluated. We show that the TCN outperforms the LSTM, achieving an accuracy of 93.5% to recognize 5+1 gestures with a model size of only 36 KB. We demonstrate our contributions on a wrist-worn full prototype including a display for enhanced human-computer interaction.