Visible-Infrared Person Re-identification (VI-ReID) aims to match person images of the same identity across different modalities, presenting challenges due to significant intra- and cross-modal variations. Existing methods primarily focus on extracting features from original modalities and projecting them into a unified space, using cosine similarity of output features for matching, while overlooking two critical aspects: 1) the identity-aware clues embedded in human poses during irregular movements, and 2) the role of intra-modal latent information in cross-modal identity association. To address these limitations, we propose a Pose-Guided Cross-modal Invariant Feature Learning (PG-CIFL) framework that leverages pose priors and excavates intra-modal useful information. Specifically, we first train a pose-guided cross-modal feature fusion teacher model using pairwise datasets. This model then extracts pose-aware person features through distiller apparatus that containing rich cross-modal information for the ReID network. The features are processed through a dual-branch network to obtain both intra-modal and cross-modal characteristics. Finally, we optimize identity inference by combining these features with mined latent intra- and inter-modal relationships. Comprehensive experiments on SYSU-MM01 and RegDB datasets demonstrate the superior performance of our PG-CIFL framework.