Identifying and classifying defects in semiconductor wafer maps is a critical yet challenging task due to the presence of subtle, diverse, and imbalanced defect patterns. Existing methods often fail to accurately detect tiny defects or capture complex global patterns in wafer maps. Unlike prior sequential CNN-ViT pipelines, this study introduces a parallel hybrid deep learning framework that integrates a lightweight Convolutional Neural Network (CNN) with a compact Vision Transformer (ViT) to address these complexities in the WM-811K wafer map dataset. The model’s robust preprocessing pipeline, which includes histogram equalization and advanced data augmentation, enhances the visibility of fine-grained and rare defects. By fusing features from both local (CNN) and global (ViT) branches and applying dimensionality reduction followed by Support Vector Machine (SVM) classification, the system achieves ~94% test accuracy. Comparative analysis demonstrates the effectiveness of the hybrid architecture in detecting rare defects and handling imbalanced data. The proposed method provides a robust and efficient solution for automated wafer inspection, suitable for real-time semiconductor manufacturing environments.