This paper presents a design and implementation of a generic hand-gestation-based input modality using computer vision. It can be used in a multi-modal human–computer interaction (HCI) platform to control any HCI application through its customizable action-activity mapping. In the proposed approach, we have investigated the potential for an additional secondary modality in coexistence with the existing primary modalities such as keyboard, mouse, and microphone used in generic HCI. The proposed approach, inspired by natural human interaction, can reduce the cognitive load on the user during HCI. We have proposed a unique set of 16 hand gestures, which we have gathered using a polling process. We have used several techniques, such as SVM and LSTM with MediaPipe, to recognize those 16 hand gestures by considering them in the form of static and dynamic hand gestures. A dataset of 2,000 samples per gesture was collected for static gestures, which led to an accuracy of 98% using SVM. Similarly, 1950 samples for dynamic gestures, each consisting of 20 frames, were used to achieve a precision of 97. 82% with LSTM. The combination of these two models resulted in a system capable of controlling various applications (such as YouTube, Facebook, LinkedIn and Gmail), which was evaluated through a survey involving 30 participants. These participants, including researchers, undergraduate students, and parents of disabled individuals (some of whom had actual or simulated disabilities), provided feedback on the system’s usefulness and usability.