In the domain of human action recognition (HAR), infrared data has emerged as a pivotal sensing technology in robotic applications due to its robust performance under low-light or rapidly changing lighting conditions. However, HAR methods that depend exclusively on infrared data fall short due to the lack of color and texture features. Combining other modal data, such as RGB modality, can alleviate this problem. In this paper, we introduce a cross-modal knowledge distillation approach that achieves knowledge transfer by using a teacher network with RGB data input to guide the recognition of an infrared data student network. The RGB data is exclusively utilized during the training phase. To make full use of RGB information, firstly, we construct a multi-scale graph cross-attention module between different convolutional layers of the teacher and student networks to reduce the modality difference between infrared data and RGB data modalities. Secondly, we employ a decoupled knowledge distillation module (DKD) to focus on more dark knowledge, i.e., knowledge related to similar behaviors, thereby enhancing the network’s robustness. We prove the effectiveness of the proposed approach on two datasets, i.e., NTU RGB+D and PKU-MMD datasets, providing strong support for the intelligent behavior of robots in various environments.