<p>Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are widely used in multimodal hybrid brain-computer interface (BCI) systems. The complementary advantages of these two modalities of neuroimaging data can effectively enhance the performance of BCI systems. However, the current multimodal study is still insufficient in the utilization efficiency of the bimodality. In this paper, we propose a dual-modal parallel network (DMPNet), which integrates EEG temporal features and fNIRS spatial features through a dual-branch parallel structure for cognitive task classification. The method combines adaptive multi-scale temporal convolution and temporal attention mechanisms to capture EEG temporal features, while introducing deep spatial convolution and spatial attention mechanisms to extract fNIRS spatial information. This dual-branch parallel model simultaneously handles two modalities of data and achieves effective complementarity fusion of information. The proposed method was tested on publicly available cognitive dataset, including NBack task, motor imagery (MI) task, and mental arithmetic (MA) task. In fivefold cross-validation experiments, DMPNet achieves the best average accuracy of 90.21%, 85.82%, and 94.94% in N-Back, MI, and MA, respectively. The results demonstrate that the method outperforms existing techniques and presents considerable generalization capabilities.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DMPNet: dual-modal parallel network with spatiotemporal fusion for cognitive task classification

  • Manqing Wang,
  • Jiacheng Gu,
  • Lin Yang,
  • Hongli Chang,
  • Dongrui Gao

摘要

Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are widely used in multimodal hybrid brain-computer interface (BCI) systems. The complementary advantages of these two modalities of neuroimaging data can effectively enhance the performance of BCI systems. However, the current multimodal study is still insufficient in the utilization efficiency of the bimodality. In this paper, we propose a dual-modal parallel network (DMPNet), which integrates EEG temporal features and fNIRS spatial features through a dual-branch parallel structure for cognitive task classification. The method combines adaptive multi-scale temporal convolution and temporal attention mechanisms to capture EEG temporal features, while introducing deep spatial convolution and spatial attention mechanisms to extract fNIRS spatial information. This dual-branch parallel model simultaneously handles two modalities of data and achieves effective complementarity fusion of information. The proposed method was tested on publicly available cognitive dataset, including NBack task, motor imagery (MI) task, and mental arithmetic (MA) task. In fivefold cross-validation experiments, DMPNet achieves the best average accuracy of 90.21%, 85.82%, and 94.94% in N-Back, MI, and MA, respectively. The results demonstrate that the method outperforms existing techniques and presents considerable generalization capabilities.