Vision Transformers (ViTs) present state-of-the-art performance on various tasks in modern computer vision. However, the black-box nature of ViTs limits their adoption in safety-critical applications where interpretability and explainability are crucial. Although several post hoc explainability methods exist, they often struggle with generalizability across different architectures and input modalities. Inherently explainable methods, which directly link model predictions to interpretable explanations, remain less explored. In this work, we propose SimEx-ViT, a novel framework for enhancing the explainability of attention mechanisms in Vision Transformers. SimEx-ViT introduces a novel learnable explainability matrix that integrates directly into the attention mechanism, enabling interpretable behavior in transformer models. This matrix is composed of trainable parameters that dynamically influence attention weights across heads. To guide the emergence of interpretable patterns, we introduce auxiliary loss functions: an entropy-based loss that encourages confident and focused attention distributions, and a diversity-promoting loss based on Jensen-Shannon divergence to ensure varied representational learning across different attention heads. We also propose a loss to guide the attention weights based on segmentation masks for semantic segmentation. By jointly optimizing these components, SimEx-ViT ensures that the learned representations are both interpretable and semantically rich without compromising model performance. Experimental results demonstrate that our method achieves a mean Intersection over Union (mIoU) of 53.29%, significantly outperforming ViT (41.42%) and eX-ViT (42.03%) when evaluating attention maps against ground truth segmentation masks. SimEx-ViT represents a step forward toward transparent and trustworthy Vision Transformers, enabling their safer deployment in critical applications such as autonomous driving, healthcare, and defense.