Tongue diagnosis is a serious non-invasive technique used in both traditional and modern medical diagnosis to examine the internal health status. Manual interpretation however is subjective and suffers inconsistencies. In this regard, paper introduce TongueViT, a new Vision Transformer (ViT)-based system of automated classification of tongue images. Compared to traditional CNN-based methods, TongueViT with the self-attention mechanism of transformers can exploit global contextual features to discriminate accurately between affected and normal tongue images. This model was trained and tested on a preprocessed dataset containing two categories of affected and healthy with stupendous classification accuracy of 99%. The results shown that TongueViT massively outcompetes the traditional models both in terms of performance and interpretability. Attention maps also demonstrate that the model looks at clinically interesting areas of the tongue, making it more probable to be translated into real-world diagnostic assistance. These results confirm the usefulness of transformer-based models in biomedical image analysis and open the prospect of AI-aided tongue diagnostics in telemedicine and clinical screening.