Single-modal (e.g., visual) lip recognition is susceptible to uneven illumination, noise interference and other factors, resulting in limited recognition performance. In this paper, we propose a Chinese lip recognition method based on multimodal fusion, which combines the complementary information of video and audio modalities to improve the robustness of recognition in complex environments. The model adopts a two-branch encoder architecture: the video encoder extracts spatio-temporal features of lips via 3D convolutional network and Transformer, and the audio encoder extracts spectral features of speech via 1D convolutional network, and realizes cross-modal feature fusion via long-short time memory network (LSTM). Experiments on the LRW-1000 dataset show that the proposed method achieves 86.7% accuracy in the Chinese lip recognition task, which is a 9.2% improvement over the unimodal method.