<p>Multimodal emotion recognition (MER) relies on effective fusion of heterogeneous data but is challenged by uncertain data and high-conflict evidence. Dempster–Shafer (D–S) evidence theory is advantageous for fusing uncertain information; yet, its performance is limited by two key issues: unreliable basic probability assignment (BPA) generation and counterintuitive results under high-conflict scenarios. In this paper, we propose a decision fusion framework that addresses these challenges by designing a BPA allocation method and improving the evidence theory fusion algorithm. The proposed method utilizes Possibilistic C-Means (PCM) clustering technology to construct noise-resistant interval models, enabling robust BPA generation even with small sample sizes. It also calculates evidence weights by considering both direct and indirect relationships between evidence, thereby improving the fusion accuracy of highly conflicting data. Furthermore, its modular architecture facilitates parallel execution across modalities and evidence sources, ensuring low computational cost and scalability for distributed and high-performance computing environments. We validated our method using the CMU-MOSI and CMU-MOSEI dataset and demonstrated its superior performance compared to existing methods. We also designed BPA experiments with noisy datasets and high-conflict data fusion experiments to verify the effectiveness of the proposed method.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Improved evidence theory based on information fusion for multimodal emotion recognition

  • Kejiang Xiao,
  • Wenqi Yang,
  • Guangjie Zhu,
  • Jiefan Qiu,
  • Shiyan Pang,
  • Chongming Zhao

摘要

Multimodal emotion recognition (MER) relies on effective fusion of heterogeneous data but is challenged by uncertain data and high-conflict evidence. Dempster–Shafer (D–S) evidence theory is advantageous for fusing uncertain information; yet, its performance is limited by two key issues: unreliable basic probability assignment (BPA) generation and counterintuitive results under high-conflict scenarios. In this paper, we propose a decision fusion framework that addresses these challenges by designing a BPA allocation method and improving the evidence theory fusion algorithm. The proposed method utilizes Possibilistic C-Means (PCM) clustering technology to construct noise-resistant interval models, enabling robust BPA generation even with small sample sizes. It also calculates evidence weights by considering both direct and indirect relationships between evidence, thereby improving the fusion accuracy of highly conflicting data. Furthermore, its modular architecture facilitates parallel execution across modalities and evidence sources, ensuring low computational cost and scalability for distributed and high-performance computing environments. We validated our method using the CMU-MOSI and CMU-MOSEI dataset and demonstrated its superior performance compared to existing methods. We also designed BPA experiments with noisy datasets and high-conflict data fusion experiments to verify the effectiveness of the proposed method.