Scholarlink

<p>Automated stress detection holds significant potential for healthcare and well-being monitoring, but the black-box nature of multimodal machine learning models often limits their practical adoption. This study presents a comprehensive explainable AI (XAI) analysis of multimodal stress detection systems using the StressID dataset, which incorporates synchronized physiological, video, and audio modalities. Through systematic application of SHAP, LIME, and Permutation Importance methods, we unravel the decision-making processes of both unimodal and multimodal classifiers. Our analysis reveals crucial insights: audio and video modalities demonstrate strong structural complementarity, capturing synergistic aspects of stress expression through prosodic features and facial dynamics, while physiological signals, though informative, show higher sensitivity to noise and individual variability. We show that decision-level fusion regularly outperforms feature-level techniques, preserving modality-specific expertise and avoiding the curse of dimensionality. Among fusion strategies, the Average and Sum rules provide optimal robustness, balancing contributions across modalities without requiring precise calibration. Modality-level Shapley analysis under the Average rule indicates near-equilibrium contributions across modalities (Audio: 0.064, Video: 0.062, Physio: 0.061), supporting robust evidence integration without single-modality dominance. The XAI framework not only enhances model transparency but also provides actionable guidance for system optimization, suggesting prioritized investment in audio-video quality and adaptive fusion weighting. This work advances multimodal affective computing by bridging performance and interpretability, offering both scientific insights into stress manifestation and practical architectural recommendations for trustworthy stress-monitoring systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Explainable AI for multimodal stress detection: interpreting model decisions across physiological, video and audio modalities

  • 期刊论文

  • 发表日期：

    2026年5月20日

查看全文
  • Andrea Francesco Abate,
  • Carmen Bisogni,
  • Aniello Castiglione,
  • Maddalena Migliaccio

摘要

Automated stress detection holds significant potential for healthcare and well-being monitoring, but the black-box nature of multimodal machine learning models often limits their practical adoption. This study presents a comprehensive explainable AI (XAI) analysis of multimodal stress detection systems using the StressID dataset, which incorporates synchronized physiological, video, and audio modalities. Through systematic application of SHAP, LIME, and Permutation Importance methods, we unravel the decision-making processes of both unimodal and multimodal classifiers. Our analysis reveals crucial insights: audio and video modalities demonstrate strong structural complementarity, capturing synergistic aspects of stress expression through prosodic features and facial dynamics, while physiological signals, though informative, show higher sensitivity to noise and individual variability. We show that decision-level fusion regularly outperforms feature-level techniques, preserving modality-specific expertise and avoiding the curse of dimensionality. Among fusion strategies, the Average and Sum rules provide optimal robustness, balancing contributions across modalities without requiring precise calibration. Modality-level Shapley analysis under the Average rule indicates near-equilibrium contributions across modalities (Audio: 0.064, Video: 0.062, Physio: 0.061), supporting robust evidence integration without single-modality dominance. The XAI framework not only enhances model transparency but also provides actionable guidance for system optimization, suggesting prioritized investment in audio-video quality and adaptive fusion weighting. This work advances multimodal affective computing by bridging performance and interpretability, offering both scientific insights into stress manifestation and practical architectural recommendations for trustworthy stress-monitoring systems.