Multimodal large language models (MLLMs) for medical imaging currently operate through fast pattern recognition, and do not mirror the deliberate reasoning process of expert human radiologists. We introduce HERMES (Hierarchical Energy-based Reasoning for MEdical Scans), a novel approach that builds upon Merlin’s pre-trained VLM to perform zero-shot classification of 10 common abdominal CT findings. Our energy-based framework implements hierarchical reasoning through three levels of analysis (global \(\rightarrow \) regional \(\rightarrow \) focal), with adaptive early stopping that reduces computation inherently, with similar to improved classification performance. The model learns task-specific projections and energy thresholds that enable confident cases to terminate early while complex multi-pathology cases receive deeper scrutiny—computationally mirroring radiologist workflow. HERMES enables transparent and interpretable reasoning, for practical trustworthiness of clinical MLLMs.