Traditional positional encoding (PE) methods in Vision Transformers (ViT) focus primarily on spatial information, but they may not adequately capture the complex geometric patterns intrinsic to medical images. To address this limitation, we have previously proposed a similarity-based positional encoding combining convolution operations and standard cosine similarity between image patches. In the present work we compare similarity-based PE with two traditional alternatives in ViT such as standard learned PE and rotatory PE. The goal is to show that, in addition to provide better classification accuracy of 2D images in different medical domains, the attention maps generated by similarity-based PE appears to be more meaningful than those generated by alternative encodings, focusing on the medical relevant part of the images. Finally, we also show the benefits of the proposed approach in dealing with 3D medical images, again in terms of classification performance. We validate our method on a set of six medical imaging datasets from MedMNIST which are benchmark datasets of medical images of various kinds, such as X-rays, histological samples, dermoscopic, ultrasounds and microscope images.