The study of gesture recognition is closely related to the harmonious development of human-computer interaction, which has important research significance. Aiming at the problems of long detection time and low recognition efficiency of traditional gesture detection algorithm, this paper proposes a gesture recognition model based on the combination of improved the YOLOV3 network and the Bayes classifier. The spatial transformer network is used to improve the YOLOV3 network for processing gesture information and extract key gesture features, so as to solve the problem of data vulnerability while maintaining the depth extraction of feature information. Then, the features are input into the combined model of PCA network and Bayes classifier for predicting gesture categories with reducing the dimension of data and improve the classification accuracy. Finally, the comparison test is performed using the public and self-made dataset, and experimental result illustrate that the propose algorithm can improve detection accuracy with higher effectiveness.