Vision transformer (ViT) architectures use self-attention mechanisms to capture the overall context of images. They often outperform convolutional neural networks (CNNs) in certain tasks. However, their ability to resist adversarial attacks is less studied than that of CNNs. This paper assesses the adversarial strength of vanilla ViTs, hybrid ViTs, and CNNs against various attack scenarios. We performed experiments on 1000 ImageNet-1k images with \(L_p\) -norm attacks ( \(L_0\) , \(L_1\) , \(L_2\) , and \(L_\infty\) ), preprocessing-based defenses, and adaptive attacks based on expectation over transformation (EOT) method. We analyzed the results using feature maps, attention maps, perturbation energy spectra, Grad-CAM visualizations, and image quality metrics, such as PSNR, SSIM, and MAD. Our results show that vanilla ViTs are more robust against \(L_0\) and some \(L_2/L_\infty\) attacks. They require higher-intensity perturbations spread across frequency spectra. Hybrid ViTs provide the best defense against \(L_1\) , projected gradient descent- \(L_2\) , fast gradient sign method- \(L_\infty\) , black-box, and EOT attacks due to their balanced feature processing. Interestingly, smaller vanilla ViT models, such as ViT-S-16, outperform larger models against iterative attacks. Black-box attacks created with ViTs transfer to CNNs, but not the other way around. While vanilla ViTs can resist high-frequency preprocessing-based defenses, such JPEG and spatial smoothing, hybrid ViTs benefit most from thorough preprocessing methods. These results indicate that no single architecture is perfectly robust. The choice of architecture should depend on expected threat models. ViT-based architectures mark an important step forward in defending against adversarial attacks in computer vision tasks.
