Unsupervised visible-infrared person re-identification presents significant challenges due to the huge modality gap and the absence of labels across modalities. Recent research have predominantly focused on evaluating and enhancing the quality of pseudo labels or improving cross-modality label associations. However, these approaches mostly rely on global features, which often struggle to capture fine-grained identity details and are prone to be affected by factors like pose variations, occlusions, background noise and so on, resulting in a lack of ability to distinguish between identities and mitigate inter-modality differences. To address this limitation and extract more nuanced features, we propose the Prototype-Guided Fine-Grained Feature Learning (PGFG) module. It leverages modality-shared learnable prototypes to guide the model in refining feature representations with a focus on details, concurrently achieving implicit segmentation and semantic alignment, so as to obtain fine-grained and discriminative representation across modalities. Furthermore, we introduce prototype diversity loss to promote the diversity of learnable prototypes for more comprehensive details mining. Extensive experimental validation demonstrates that the resulting feature representations significantly enhance intra-class inter-modality connectivity, substantiating the effectiveness of our proposed method. Compared with the mainstream unsupervised works, our method shows superior performance under various settings.