While transformer-based models dominate NLP and vision applications, their underlying mechanisms to map input space to label space semantically are not well understood. We study the sources of representation vulnerabilities in vision transformers (ViT), where perceptually identical images have very different representations and semantically unrelated images have identical representations. Our analysis reveals that imperceptible input changes result in significant representation changes, particularly in later layers, suggesting performance instabilities. Our comprehensive study shows adversarial effects, while subtle in early layers, propagate and amplify through the network, becoming most pronounced in middle to late layers. This insight motivates NeuroShield-ViT, a novel defense mechanism that strategically neutralizes vulnerable neurons in earlier layers to prevent the cascade of adversarial effects. We demonstrate NeuroShield-ViT’s effectiveness across various attacks, particularly against strong iterative attacks, and showcase its remarkable zero-shot generalization capabilities. Without fine-tuning, our method achieves 77.8% accuracy on adversarial examples, surpassing conventional robustness methods. Our results shed new light on adversarial effect propagation through ViT layers while providing a promising approach to enhance vision transformer robustness against adversarial attacks.