A Multimodal Explainable AI Framework for Interpreting Image Classifiers
摘要
Explainable Artificial Intelligence (XAI) aims to develop transparency in AI models, enabling easy interpretation and evaluation by users. In visual tasks, a common approach is to use saliency maps that represent which parts of an image influenced the model’s prediction. While saliency maps are widely used in image classification to highlight the most relevant pixels for a model’s decision, they often lack the context needed for complete understanding. To address this, we designed an integrated visual saliency with textual descriptions that forms a more effective and user-friendly multimodal XAI system. In this study, we present a novel black-box Multimodal Guided Input Sampling for Explanation (GuISE) XAI approach that produces integrated saliency-text explanations without relying on gradient information or external segmentation networks. This ensures that our method is both lightweight and broadly compatible. We benchmark our approach against state-of-the-art methods using images of the MS-COCO 2017 validation dataset and the MSRA-B dataset, and show that our method surpasses existing approaches.