In video-based human behavior recognition, a challenge with two-stream neural networks is the inadequate feature extraction and disregard for local features. This leads to poor performance in processing actions with high inter-class similarity. In order to overcome these problems, we propose a model for the recognition of behaviour that uses a two-stream neural network with modified multi-scale ResNet and BiGRU. To classify spatial features, we use a multi-scale ResNet structure with three parallel convolutional kernels of different sizes. This allows us to effectively capture spatially varying features during motion. In the classification of temporal features, BiGRU models are built to capture both forward and backward temporal features to obtain more comprehensive contextual information. The UCF101 dataset was used in the experiments. The proposed model achieved 97.1% accuracy on the public dataset. It outperformed other algorithms in extracting feature information from similar actions, with an accuracy rate of 93.2%. This shows the superiority of the proposed model in obtaining feature information.