Multimodality in Explanatory Interactions
摘要
Studies on human–human interaction have shown that multimodal signals, such as hand gestures, facial expressions, and prosody, are important for both cognitive and interactive processes of meaning-making. Multimodal behavior also plays a crucial role in the area of knowledge transfer, for which explanations are central. Recent studies of multimodal behavior in explanations within the framework of the global structure of explanatory interactions suggest that it can be understood as interdependent and mutually informing. Therefore, this chapter first provides an overview over conceptualizations of explanations, focusing on approaches that view explanations as a social process, thereby emphasizing their co-constructive, interactive nature. A model is introduced that provides an empirically based, co-constructed global structure of explanatory interactions. The main topic of the second section is the forms and functions of multimodal behaviors. The focus is on prosody and gestures, examining how they support knowledge transfer and understanding by structuring speech, creating coherence, emphasizing important information, and, in the case of gestures, providing additional content related to the explanandum. The chapter then suggested how both the global structure of explanatory interactions and multimodal behavior could be brought together in a heuristic for more detailed further research and future fine-tuned implementation of multimodal behavior in sXAI.