Scholarlink

<p>Micro-expression (ME), as an involuntary and brief facial expression, holds significant potential applications in fields such as political psychology, lie detection, law enforcement, and healthcare. Most existing micro-expression spotting (MES) methods predominantly learn from optical flow features while neglecting the detailed information contained in RGB images. To address this issue, this paper proposes a multi-scale hierarchical semantic-guided end-to-end multimodal fusion framework based on Convolutional Neural Network (CNN)-Transformer for MES, named MESFusion. Specifically, to obtain cross-modal complementary information, this scheme sequentially constructs a Multi-Scale Feature Extraction Module (MFEM) and a Multi-scale hierarchical Semantic-Guided Fusion Module (MSGFM). By introducing an Optical Flow-Driven fusion feature Integration Module (OF-DIM), the correlation of non-scale fusion features is modeled in the channel dimension. Moreover, guided by the optical flow motion information, this approach can adaptively focus on facial motion areas and filter out interference information in cross-modal fusion. Extensive experiments conducted on the CAS(ME)<sup>2</sup> dataset and the SAMM Long Videos dataset demonstrate that the MESFusion model surpasses competitive baselines and achieves new state-of-the-art results.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Micro-expression spotting based on multi-modal hierarchical semantic guided deep fusion and optical flow driven feature integration

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年4月11日

查看全文
  • Haolin Chang,
  • Zhihua Xie,
  • Fan Yang

摘要

Micro-expression (ME), as an involuntary and brief facial expression, holds significant potential applications in fields such as political psychology, lie detection, law enforcement, and healthcare. Most existing micro-expression spotting (MES) methods predominantly learn from optical flow features while neglecting the detailed information contained in RGB images. To address this issue, this paper proposes a multi-scale hierarchical semantic-guided end-to-end multimodal fusion framework based on Convolutional Neural Network (CNN)-Transformer for MES, named MESFusion. Specifically, to obtain cross-modal complementary information, this scheme sequentially constructs a Multi-Scale Feature Extraction Module (MFEM) and a Multi-scale hierarchical Semantic-Guided Fusion Module (MSGFM). By introducing an Optical Flow-Driven fusion feature Integration Module (OF-DIM), the correlation of non-scale fusion features is modeled in the channel dimension. Moreover, guided by the optical flow motion information, this approach can adaptively focus on facial motion areas and filter out interference information in cross-modal fusion. Extensive experiments conducted on the CAS(ME)2 dataset and the SAMM Long Videos dataset demonstrate that the MESFusion model surpasses competitive baselines and achieves new state-of-the-art results.