Visible-infrared person re-identification (VI-ReID) aims to match person identities across day and night scenarios through cross-modal retrieval. Existing studies predominantly rely on supervised learning, which requires extensive annotated data for cross-modal alignment. This annotation process is significantly more costly than that of the single-modal ReID. Although current unsupervised methods attempt to mitigate modal discrepancies by leveraging nearest-neighbor samples or generative adversarial networks, they still face two major challenges: (1) biased pseudo-label distributions caused by noise interference, and (2) insufficient alignment of fine-grained cross-modal shared features. To address these limitations, we propose a Hierarchical Calibration Network (HCNet) that progressively optimizes both pseudo-label generation and cross-modal feature alignment. Specifically, we first design a Median-enhanced Feature Calibration (MFC) module that suppresses feature noise by generating channel-wise attention weights through hybrid median-average-max pooling operations. Then, we propose a Multi-scale Asymmetric Contrast (MAC) module to align local person features via multi-scale asymmetric convolutions and modal contrastive learning. Finally, we introduce a Memory-guided Prototype Refinement (MPR) module maintains stores cross-modal class prototypes in a memory bank and leverages these prototypes to iteratively refine pseudo-labels. Extensive experiments on two benchmark datasets demonstrate the effectiveness of the proposed method against state-of-the-art methods.