Ophthalmologists often rely on multimodal data to improve diagnostic precision. However, data on complete modalities are rare in real applications due to a lack of medical equipment and data privacy concerns. Traditional deep learning approaches usually solve these problems by learning representations in latent space. However, we highlight two critical limitations of these current approaches: (i) Task-irrelevant redundant information existing in complex modalities (e.g., massive slices) leads to a significant amount of redundancy in latent space representations. (ii) Overlapping multimodal representations make it challenging to extract features that are unique to each modality. To address these, we introduce the Essence-Point and Disentangle Representation Learning (EDRL) strategy that integrates a self-distillation mechanism into an end-to-end framework to enhance feature selection and disentanglement for robust multimodal learning. Specifically, Essence-Point Representation Learning module selects discriminative features that enhance disease grading performance. Moreover, the Disentangled Representation Learning module separates multimodal data into modality-common and modality-unique representations, reducing feature entanglement and enhancing both robustness and interpretability in ophthalmic disease diagnosis. Experiments on ophthalmology multimodal datasets demonstrate that the proposed EDRL strategy outperforms the state-of-the-art methods significantly. Code is available at GitHub Repository .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation

  • Xinkun Wang,
  • Yifang Wang,
  • Senwei Liang,
  • Feilong Tang,
  • Chengzhi Liu,
  • Ming Hu,
  • Chao Hu,
  • Junjun He,
  • Zongyuan Ge,
  • Imran Razzak

摘要

Ophthalmologists often rely on multimodal data to improve diagnostic precision. However, data on complete modalities are rare in real applications due to a lack of medical equipment and data privacy concerns. Traditional deep learning approaches usually solve these problems by learning representations in latent space. However, we highlight two critical limitations of these current approaches: (i) Task-irrelevant redundant information existing in complex modalities (e.g., massive slices) leads to a significant amount of redundancy in latent space representations. (ii) Overlapping multimodal representations make it challenging to extract features that are unique to each modality. To address these, we introduce the Essence-Point and Disentangle Representation Learning (EDRL) strategy that integrates a self-distillation mechanism into an end-to-end framework to enhance feature selection and disentanglement for robust multimodal learning. Specifically, Essence-Point Representation Learning module selects discriminative features that enhance disease grading performance. Moreover, the Disentangled Representation Learning module separates multimodal data into modality-common and modality-unique representations, reducing feature entanglement and enhancing both robustness and interpretability in ophthalmic disease diagnosis. Experiments on ophthalmology multimodal datasets demonstrate that the proposed EDRL strategy outperforms the state-of-the-art methods significantly. Code is available at GitHub Repository .