Aiming at the problems of low structural recognition rate, many parameters and poor robustness of gesture sequence recognition model, this paper studies the improvement of gesture depth and skeleton feature information extraction algorithm and deep fusion gesture classification strategy and implementation network. Firstly, the original depth image is input into the SA-ConvLSTM model after gray dimension reduction transformation to extract spatio-temporal features, so as to form a sub-network for depth data extraction. Secondly, the two-dimensional skeleton data sub-network is formed by the fusion of multi-head self-attention and Bi-LSTM network, and the spatio-temporal features in depth and two-dimensional skeleton data are extracted in parallel. Finally, the two types of features are input into the multi-data fusion network to form multi-source heterogeneous gesture features. Experiments are conducted on the public dataset DHG-14/28, and the accuracy of 14 gestures can reach 89.68%. Compared with the Ensemble of Models method and MFA-Net method, the recognition rate of rough gesture and fine gesture is increased by 1.08% and 2.76%, respectively.