AI systems in the medical domain have advanced significantly, yet their adoption in clinical practice remains limited partly due to their “black box” nature. To address this challenge, we conducted a comparative user-centric analysis of visual, textual, and multimodal explainable artificial intelligence (XAI) methods for medical image diagnosis. We evaluated eight distinct XAI approaches across four key dimensions: understandability, completeness, speed, and applicability. Our survey of 33 physicians from various specialties revealed that 88% agree that AI explanations are important for diagnosis—64% strongly agree. Statistical analysis showed that a combination of bounding box and report achieved highest ratings across all four evaluation dimensions, significantly outperforming 2nd-best methods in all dimensions (p < 0.05). Visual methods alone, particularly heatmaps, often performed worse than no explanation, while report was preferred over interactive chatbot among textual methods. Concerning our analysis of incorrect AI predictions, we found that 50% of participants made incorrect or partially incorrect diagnoses when presented with false AI suggestions, highlighting significant risks of automation bias. These findings provide important insights for developing clinically effective XAI systems that complement physician expertise while maintaining appropriate levels of trust.