Explainable Artificial Intelligence (XAI) aims to develop transparency in AI models, enabling easy interpretation and evaluation by users. In visual tasks, a common approach is to use saliency maps that represent which parts of an image influenced the model’s prediction. While saliency maps are widely used in image classification to highlight the most relevant pixels for a model’s decision, they often lack the context needed for complete understanding. To address this, we designed an integrated visual saliency with textual descriptions that forms a more effective and user-friendly multimodal XAI system. In this study, we present a novel black-box Multimodal Guided Input Sampling for Explanation (GuISE) XAI approach that produces integrated saliency-text explanations without relying on gradient information or external segmentation networks. This ensures that our method is both lightweight and broadly compatible. We benchmark our approach against state-of-the-art methods using images of the MS-COCO 2017 validation dataset and the MSRA-B dataset, and show that our method surpasses existing approaches.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multimodal Explainable AI Framework for Interpreting Image Classifiers

  • Arju Bano,
  • Monidipa Das,
  • Rajendra Pamula

摘要

Explainable Artificial Intelligence (XAI) aims to develop transparency in AI models, enabling easy interpretation and evaluation by users. In visual tasks, a common approach is to use saliency maps that represent which parts of an image influenced the model’s prediction. While saliency maps are widely used in image classification to highlight the most relevant pixels for a model’s decision, they often lack the context needed for complete understanding. To address this, we designed an integrated visual saliency with textual descriptions that forms a more effective and user-friendly multimodal XAI system. In this study, we present a novel black-box Multimodal Guided Input Sampling for Explanation (GuISE) XAI approach that produces integrated saliency-text explanations without relying on gradient information or external segmentation networks. This ensures that our method is both lightweight and broadly compatible. We benchmark our approach against state-of-the-art methods using images of the MS-COCO 2017 validation dataset and the MSRA-B dataset, and show that our method surpasses existing approaches.