Medical imaging often suffers from limited labeled data and substantial domain gaps when transferring models pre-trained on general-purpose benchmarks such as ImageNet. This study systematically compares three training strategies for Vision Transformers (ViTs) on a four-class retinal Optical Coherence Tomography (OCT) dataset(CNV, DME, Drusen, Normal): (1) training from scratch, (2) conventional ImageNet-based pre-training, and (3) a novel domain-specific pre-training method using OCT breast cancer images (adipose tissue vs. cancer). Experimental results clearly show that the domain-specific OCT breast pre-training significantly improves classification accuracy compared to both ImageNet pre-training and training from scratch, particularly under limited-data scenarios. These findings challenge the prevailing view that general-domain pre-training has limited utility in medical imaging, instead emphasizing the essential role of domain alignment in pre-training datasets. Our results highlight the critical advantage of domain-specific pre-training in medical imaging AI, demonstrating improved accuracy and potential for earlier retinal disease detection even with scarce labeled data. Future research should focus on constructing larger OCT-specific pre-training datasets and exploring advanced self-supervised methods tailored explicitly for medical imaging tasks.