Depression is a serious mental health disorder posing significant challenges to global health, particularly on vulnerable populations such as the elderly. However, the performance of current multimodal depression detection approaches remains constrained by both data scarcity and architectural limitations in capturing long-range dependencies and integrating heterogeneous modalities. To address the issue, we propose a hierarchical fusion framework comprising two key stages: DepMamba-based multimodal Perception Fusion with personalized features and hierarchical ensemble Prediction Fusion with top-level supervision. Firstly, we leverage DepMamba to perceive depression-related cues and integrate cross-modal information. In addition, we utilize large language models to perceive and incorporate personalized individual characteristics into the detection process, thereby accounting for the nuanced variations in depressive symptoms. And then we adopt hierarchical ensemble guided by binary depression labels to refine fine-grained prediction, forming a closed-loop structure for robust detection. Additionally, we design a multilevel data augmentation strategy spanning both utterance and individual levels, and incorporate pseudo-labeling to enrich scarce training data and enhance generalization. The proposed method is evaluated on the MPDD-Elderly dataset, demonstrating strong performance with a top accuracy of 84.48%, an average accuracy of 84.16%, and an F1 score of 84.79%. The present work highlights the effectiveness of hierarchical fusion and top-level supervision for fine-grained depression detection with subtle personalized factors.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Hierarchical Fusion Modeling from Perception to Prediction with Personalized Features for Multimodal Depression Detection

  • Yingying Zhou,
  • Han Chi,
  • Jingyao Xue,
  • Yiming Gao,
  • Minchi Hu,
  • Yuhua Wen,
  • Qifei Li,
  • Yingming Gao,
  • Ya Li

摘要

Depression is a serious mental health disorder posing significant challenges to global health, particularly on vulnerable populations such as the elderly. However, the performance of current multimodal depression detection approaches remains constrained by both data scarcity and architectural limitations in capturing long-range dependencies and integrating heterogeneous modalities. To address the issue, we propose a hierarchical fusion framework comprising two key stages: DepMamba-based multimodal Perception Fusion with personalized features and hierarchical ensemble Prediction Fusion with top-level supervision. Firstly, we leverage DepMamba to perceive depression-related cues and integrate cross-modal information. In addition, we utilize large language models to perceive and incorporate personalized individual characteristics into the detection process, thereby accounting for the nuanced variations in depressive symptoms. And then we adopt hierarchical ensemble guided by binary depression labels to refine fine-grained prediction, forming a closed-loop structure for robust detection. Additionally, we design a multilevel data augmentation strategy spanning both utterance and individual levels, and incorporate pseudo-labeling to enrich scarce training data and enhance generalization. The proposed method is evaluated on the MPDD-Elderly dataset, demonstrating strong performance with a top accuracy of 84.48%, an average accuracy of 84.16%, and an F1 score of 84.79%. The present work highlights the effectiveness of hierarchical fusion and top-level supervision for fine-grained depression detection with subtle personalized factors.