This study examines the feasibility and performance of using single OCT slices from the OCTA-500 dataset to classify DR (Diabetic Retinopathy) and AMD (Age-Related Macular Degeneration) with a pre-trained transformer-based model (RETFound). The experiments revealed the effective adaptation capability of the pretrained model to the retinal disease classification problem. We further explored the impact of using different slices from the OCT volume, assessing the sensitivity of the results to the choice of a single slice (e.g., “middle slice”) and whether analyzing both horizontal and vertical cross-sectional slices could improve outcomes. However, deep neural networks are complex systems that do not indicate directly whether they have learned and generalized the disease appearance as human experts do. The original dataset lacked disease localization annotations. Therefore, we collected new disease classification and localization annotations from independent experts for a subset of OCTA-500 images. We compared RETFound’s explainability-based localization outputs with these newly collected annotations and found that the region attributions aligned well with the expert annotations. Additionally, we assessed the agreement and variability between experts and RETFound in classifying disease conditions. The Kappa values, ranging from 0.35 to 0.69, indicated moderate agreement among experts and between the experts and the model. The transformer-based RETFound model using single or multiple OCT slices, is an efficient approach to diagnosing AMD and DR.