<p>Graph Convolutional Networks (GCNs), owing to their powerful capability in modeling graph structures, have been widely applied to skeleton-based action recognition and have demonstrated excellent performance. However, mainstream GNN methods are often constrained by relatively flat architectures, which hinder their ability to fully capture and represent the hierarchical features inherent in human actions. To address this limitation, this paper proposes a Second-Order Hierarchical Graph Convolutional Network (SOH-GCN), designed to more effectively model and integrate hierarchical feature information in human actions. Specifically, in order to capture dynamic features at different levels, a Hierarchical Graph Convolutional Network (H-GCN) architecture is first designed to perform graph convolutions on the full-body graph, key half-body graph, and key limb graph respectively. Next, aiming to better capture complex interdependencies among hierarchical features and enable deeper integration, a Second-Order Feature Fusion mechanism is designed based on global covariance pooling. Furthermore, a novel loss function named Cross-Channel Dual-Pooling Loss (CCDP-Loss) is proposed to establish explicit associations between the hierarchical channel representations and specific action categories, thereby enhancing the model’s ability to distinguish subtle differences between action classes. Extensive experiments demonstrate that our proposed method achieves superior performance on three widely used benchmark datasets: NTU RGB+D, NTU RGB+D 120, and NW-UCLA.</p>