Integrating image and text data through multi-modal learning has gained significant attention in medical imaging research, building on its success in computer vision. While substantial progress has been made in developing medical foundation models and enabling their zero-shot transfer to downstream tasks, the potential of test-time adaptation remains largely underexplored in this domain. Inspired by recent advances in transductive learning and parameter-efficient fine-tuning, we investigate test-time adaptation of medical vision-language models. Our method leverages information-theoretic principles, maximizing the mutual information between the visual inputs and the text-based class representations, while minimizing a Kullback-Leibler divergence term penalizing deviation of the predictions from the zero-shot outputs. Building on this foundation, we introduce the first structured benchmark for test-time adaptation of medical vision-language models, exploring strategies tailored to the unique challenges of medical imaging. Our extensive experiments include two medical modalities, three specialized foundation models, six downstream tasks, and multiple state-of-the-art test-time adaptation methods, demonstrating significant performance improvements and establishing a new benchmark for this emerging field. The code is available at: https://github.com/FereshteShakeri/TTAMedVLMs .