Theoretical Aspects of Multimodal Processing
摘要
The concept of sXAI is inspired by (human) social interaction, the primary case of which is face-to-face communication involving not only speech and language but also nonverbal modalities such as facial expression, gaze, gesture, head movement, and body posture. Such multimodal communication facilitates language processing in humans and becomes relevant for sXAI systems when they are (embodied) communicating agents able to produce multimodal behaviors and understand those of human users. In this chapter, we describe theoretical aspects of multimodal processing in human interaction (binding and segregation of modalities, how multimodal behavior is produced and understood) and their relation to computational models of multimodal interaction for conversational agents.