In recent years, there have been significant improvements in lip-reading systems, leading to breakthroughs in accuracy in certain specific scenarios. However, most existing algorithms rely on audio as auxiliary features and lack research on lip-reading in silent environments. Additionally, the large-scale parameters of these algorithms are not suitable for deployment on resource-constrained mobile devices, limiting the development in certain scenarios. In this work, we propose the integration of image difference in a bimodal approach to effectively extract lip motion and variability, reducing the interference from irrelevant frames and enhancing the model’s robustness. Furthermore, in the visual frontend of the image frame modality branch, we replace the commonly used 3D CNN with pseudo-3D CNN and replace the convolution layers in 2D ResNet-18 with TCN. This approach not only improves the extraction of spatial and temporal features from image frames but also significantly reduces the parameter size. Experimental results show that our model achieves nearly the same accuracy as state-of-the-art models while reducing the parameter size by almost 9 times. It has lower computational resource requirements and can meet the demands of lip-reading scenarios on mobile devices.