Accurate identification of breast cancer laterality (left versus right) is essential for treatment planning and clinical decision support. In this study, we propose a multimodal deep learning framework that integrates paired mammographic images with patient clinical features from the TCGA-BRCA dataset. We benchmark three backbone architectures—Vision Transformer (ViT), ResNet-50, and a custom Convolutional Neural Network (CNN)—to evaluate the effectiveness of multimodal fusion. Each model processes paired cancerous and normal breast images along with structured clinical metadata such as age, disease type, treatment history, and lymph node status. The ViT achieved the highest test accuracy (98.99%, 95% CI: ± 0.4%), followed by CNN (98.80%) and ResNet-50 (95.12%). Statistical significance testing using the McNemar test confirmed that both ViT and CNN significantly outperformed ResNet-50 (p < 0.01). Error analysis revealed challenges in low-contrast cases and bilateral abnormalities. These findings demonstrate the novelty and effectiveness of combining paired image analysis with structured clinical data, highlighting the potential of multimodal transformers for robust and clinically relevant laterality prediction.