LipVision: Lip-Reading Model Using Machine Learning
摘要
Lip reading, or visual speech recognition, is the process of figuring out what someone is saying in a video by looking at how their lips move. Recent improvements in lip-reading technology have been made possible by the fast growth of deep learning methods and the availability of large lip-reading datasets. This has been a big help to the field. To make communication easier and more accessible, we need good lip-reading devices. In this study, we address this problem by introducing a new method called LipVision that combines the MediaPipe framework with a Bi-directional Gated Recurrent Units (BiGRU) architecture. The input video is first preprocessed by extracting facial landmarks using MediaPipe Face Mesh, focusing on the lip region (landmarks 216–430), which is then cropped, resized, converted to grayscale, and normalized to ensure consistent feature extraction. The resulting sequence of processed frames is fed into a 3D Convolutional Neural Network (3D CNN) to capture both spatial and temporal features, followed by Bidirectional GRU layers for modeling temporal dependencies in both directions. For the final decoding stage, we employ the Connectionist Temporal Classification (CTC) loss function, enabling end-to-end training without explicit alignment between input frames and output text sequences. According to evaluations on the GRID Corpus Dataset, the proposed model achieves an accuracy of 91.9% and MIRACL-VC1 dataset achieves an accuracy of 99.1%, surpassing the performance of previously reported methods in the literature.