Deep learning algorithms show tremendous potential for clinical decision-making - for example, in providing automated diagnoses of imaging data. However, typical clinical datasets often are limited in size, modalities, and contain heterogeneous, incomplete data, which presents challenges for deep learning frameworks that necessitate larger, uniform datasets, complicating their deployment especially with new types of disease models. In this work, we present a case study for deep learning in such a challenging setting in the context of diagnosing Asian choroidal Vasculopathy (ACV), which is a retinopathy profile currently under discussion in ophthalmology to be differentiated from age-related macular degeneration (AMD). We first introduce a novel, human-annotated multimodal dataset for ACV versus AMD diagnosis incorporating four different imaging modalities. We next explore the usefulness of “foundation models” for this data, compared to traditional dataset-specific training. Most importantly, we investigate which of the four modalities is most discriminative and whether bi-modal classification is able to enhance performance across multiple fusion approaches. We also discuss first results of salient features using explainability techniques.