In action recognition, while combining spatio-temporal videos with skeleton features can enhance recognition performance, it necessitates distinct models and balanced feature representations for cross-modal data. Addressing these challenges, we introduce the Deep Residual Vision Transformer (Deep-RVT), a novel architecture that combines the training stability and resilience provided by residual connections with the representational power of Vision Transformers in a seamless manner. The spatio-temporal dynamics required for identifying human actions in video sequences are precisely captured by Deep-RVT. Our model efficiently propagates low-level features across layers by embedding residual pathways into the Transformer blocks, which resolves the vanishing gradient issue and speeds up the training process. Our design leverages the integration of local and global features to encode subtle motion cues and spatial arrangements, which are essential for identifying a broad range of human activities. The proposed network is validated by lots of mainstream benchmarks. Many experimental results, conducted on the Penn-Action, ImageNet-1K and ImageNet-22K, show that the proposed network outperforms most state-of-the-art methods. Our code is available at https://anonymous.4open.science/r/RVT-6D6F/README.md .