Imagine a radiology department where vision-language models (VLMs) assist with report generation. For such systems to be safe, they must accurately understand spatial relationships, a skill essential for radiologists, where mistakes have led to serious consequences, such as wrong-side surgeries. In our publication [1], we show that VLMs fail at this fundamental ability. Models were asked to identify the relative position of two anatomical structures in a CT slice. Even advanced VLMs, such as GPT4o, performed only at chance level, raising concerns about their reliability in clinical routine. How could they properly describe localizations in reports without this capability? We investigated potential solutions. Since segmentation models are already highly accurate, their outputs can be used to place markers on the anatomical structures. Prior work in computer vision shows that such markers can enhance spatial reasoning. While markers yielded moderate gains, accuracy remained far below results on natural images. A deeper analysis revealed the underlying cause. VLMs already possess strong prior anatomical knowledge. In other words, they “know” where organs are typically located in standard human anatomy. Instead of analyzing the actual CT image, they often fall back on this memorized knowledge when answering spatial questions. For example, if asked whether the liver is to the right of the stomach, a model may simply respond “yes” based on general anatomy, without inspecting the image at all. This shortcut is dangerous: in cases such as situs inversus, or post-surgical alterations, where organ positions deviate from the norm, the model will confidently give incorrect answers.We release MIRP, a benchmark designed to systematically test spatial reasoning. Details on https://wolfda95.github.io/your_other_left/.