Evaluating Explainability Techniques for Machine Learning in Healthcare - A Human-Centered Approach Through Expert Interviews
摘要
Machine learning (ML) holds great promise for transforming healthcare, by enhancing diagnostic accuracy, enabling earlier interventions, and refining treatment strategies. Yet, the opacity of many ML models remains a major barrier to clinical adoption. Explainable Artificial Intelligence (XAI) is often proposed as a way to address this challenge, but we still know little about how different stakeholders perceive and engage with XAI in practice. This study explored how three XAI techniques, Local SHAP, Global SHAP, and Attention Mechanism, shape trust, usability, and decision-making across diverse user groups. We conducted semi-structured interviews with 20 participants, including clinical experts, data scientists, and members of the public, using real-world ML prediction tasks in two healthcare domains: sepsis-related mortality and psychology. Participants reviewed visualizations of model explanations and shared their perspectives in guided discussions. Through thematic analysis, five key themes emerged. Participants stressed that explainability must be paired with perceived reliability, contextual relevance, and human oversight to build trust. Local SHAP was generally seen as the most intuitive method, while Global SHAP and Attention Mechanism were often perceived as too abstract or technical. Across all groups, AI was viewed as a decision-support tool, not a replacement for clinical judgment. Successful integration of XAI was seen as dependent on adaptable explanations, integration into existing workflows, appropriate training, and interdisciplinary collaboration. Our findings highlight the need for human-centered design and institutional readiness in the development of trustworthy, usable XAI systems for healthcare.