Visible-infrared person re-identification (VI-ReID) aims to retrieve images across different modalities to match pedestrians with the same identity. Despite recent progress, challenges such as loose spatial alignment, modality discrepancies, and the entanglement of identity-unrelated confounders remain unsolved. Existing attention-based approaches often extract features from semantically misaligned regions and inadvertently encode modality-specific patterns as identity cues, impairing cross-modal generalization. To address these issues, we propose a Counterfactual Intervention Learning Network (CILNet), which incorporates causal inference to enhance the discriminability and modality invariance of learned features. Specifically, we introduce a Prototype-guided Factual Attention (PFA) module and a Prototype-guided Counterfactual Attention (PCA) module that generates identity-related factual attention and confounder-sensitive counterfactual attention using modality-shared prototypes, respectively. Counterfactual intervention is implemented by maximizing the likelihood difference between factual and counterfactual features, effectively suppressing modality-specific interference. Additionally, contrastive learning between the two feature types further disentangles identity information from confounders in the embedding space. Extensive experiments on SYSU-MM01, RegDB, and LLCM datasets demonstrate the superior performance of CILNet.