Scholarlink

<p>The accurate diagnosis of retinal pathologies is frequently impeded by the significant semantic gap and modality discrepancy between Color Fundus Photography (CFP) and Optical Coherence Tomography (OCT), particularly given the scarcity of paired clinical data. To address these challenges, CrossProto-ViT, a prototype-guided and semantically aligned multi-modal Vision Transformer framework, is presented. A hierarchical architecture is established by integrating Modality-Specific Stems and Retinal Modality Adapters within a shared ViT backbone to efficiently extract domain-specific features while preserving pre-trained knowledge. To bridge the domain gap in unpaired data, a cross-modal alignment mechanism is constructed utilizing a bidirectional Translator constrained by Conditional Optimal Transport and Cycle Consistency, coupled with an attention-based Cross-Fusion Module. Furthermore, a progressive three-stage training strategy encompassing prototype pretraining, alignment, and global optimization is implemented alongside an Adversarial Domain Discriminator to ensure robust convergence and modality invariance. Extensive experiments on heterogeneous multi-source datasets demonstrate that CrossProto-ViT achieves state-of-the-art performance, recording accuracies of 99.9% on OCT and 99.3% on CFP. The framework significantly outperforms leading baselines, including Swin Transformer V2 and the foundation model RETFound. Additionally, Grad-CAM visualizations substantiate superior lesion localization and semantic consistency, positioning CrossProto-ViT as a robust and interpretable solution for intelligent cross-modal retinal diagnosis.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

CrossProto-ViT: Unpaired Cross-Modal Prototype Alignment for Retinal Disease Diagnosis

  • 期刊论文

  • 发表日期：

    2026年3月31日

查看全文
  • Mingyang Sun,
  • Xiaoyang He

摘要

The accurate diagnosis of retinal pathologies is frequently impeded by the significant semantic gap and modality discrepancy between Color Fundus Photography (CFP) and Optical Coherence Tomography (OCT), particularly given the scarcity of paired clinical data. To address these challenges, CrossProto-ViT, a prototype-guided and semantically aligned multi-modal Vision Transformer framework, is presented. A hierarchical architecture is established by integrating Modality-Specific Stems and Retinal Modality Adapters within a shared ViT backbone to efficiently extract domain-specific features while preserving pre-trained knowledge. To bridge the domain gap in unpaired data, a cross-modal alignment mechanism is constructed utilizing a bidirectional Translator constrained by Conditional Optimal Transport and Cycle Consistency, coupled with an attention-based Cross-Fusion Module. Furthermore, a progressive three-stage training strategy encompassing prototype pretraining, alignment, and global optimization is implemented alongside an Adversarial Domain Discriminator to ensure robust convergence and modality invariance. Extensive experiments on heterogeneous multi-source datasets demonstrate that CrossProto-ViT achieves state-of-the-art performance, recording accuracies of 99.9% on OCT and 99.3% on CFP. The framework significantly outperforms leading baselines, including Swin Transformer V2 and the foundation model RETFound. Additionally, Grad-CAM visualizations substantiate superior lesion localization and semantic consistency, positioning CrossProto-ViT as a robust and interpretable solution for intelligent cross-modal retinal diagnosis.