Empowering Visually Impaired Smartphone Users: A Preliminary Study on EMG-Based Smartphone Operation with Machine Learning
摘要
Smartphones pose challenges for visually impaired users due to their reliance on touchscreens and visual feedback. While accessibility services like VoiceOver and TalkBack assist in screen navigation, they require physical contact and become cumbersome when users’ hands are occupied (e.g., holding a cane). This study proposes a non-contact alternative using electromyography (EMG) to map wrist gestures to smartphone operations. We trained a 2D convolutional neural network (CNN) on the NinaPro DB1 EMG dataset to recognize six wrist gestures corresponding to everyday touchscreen actions (e.g., screen exploration and item selection). An Android accessibility app prototype was developed to translate wrist gestures into touch inputs via ADB commands, aligning with Android Talkback accessibility logic. Preliminary user studies compare the advantages and disadvantages of touch gestures and wrist gestures, and the results show that wrist gestures have the characteristics of high usability and low learning burden. The EMG-based interaction is technically feasible as a touchless solution.