Multimodality in Agents
摘要
This chapter examines multimodality in the context of explaining AI agent behavior and reasoning. Both embodied (like robots) and unembodied (like Alexa, including text-based systems) AI systems require effective explanatory mechanisms to help users understand their functions and decisions. The analysis reveals striking similarities between design elements used for encouraging user collaboration with AI systems and those employed in creating multimodal explanations. Despite the considerable advantages offered by multimodality-based sXAI, these design choices may produce unintended negative effects on users receiving the explanations. To address this challenge, the chapter draws on philosophical research regarding emotion co-construction. By integrating methodological approaches and theoretical insights from the philosophy of science with current sXAI studies, designers can create explanations that emerge collaboratively between the explainer and explainee. Such explanations recognize how emotions influence both how humans think and how they make decisions. The path toward more effective multimodal explanations requires recognizing how users’ emotional responses shape their understanding of AI systems. Through careful design considerations, explanatory systems can acknowledge this emotional dimension while still delivering technically accurate information about AI behavior.