Objective <p>Convolutional Neural Networks (CNNs) have become essential tools for classroom student behavior recognition but lack the capability of global information capturing. In recent years, Vision Transformer (ViT) has demonstrated strong global modeling capabilities, which can be employed to strengthen the multi-level spatial information representation ability of classroom student behavior recognition models.</p> Methods <p>First, Cascaded Residual Vision Transformer (CR-ViT) model was proposed. The outputs of residual convolutional layers were integrated into multiple ViT modules to learn both shallow and deep feature representations for multi-level spatial information extraction, followed with the LSTM network for further capturing the global dependency between the sequences of ViT modules. Second, Cascaded Residual Vision Transformer with Morlet wavelet (MCR-ViT) was proposed. Based on CR-ViT, morlet wavelet transform activation layers were employed to improve the sensibility for variations of edges in feature maps.</p> Results <p>The proposed methods were validated on our collected dataset named Student Behavior in Classroom (SBIC), as well as the publicly available dataset of Student Classroom Behavior (SCB). The CR-ViT model and the MCR-ViT model improved the accuracy on SBIC by 6.10% and 7.32%, and on SCB by 0.92% and 1.14%. The MCR-ViT with second-order derivative of Morlet wavelet achieved the highest accuracy improvement compared to its variants based on other wavelet.</p> Conclusion and significance <p>both CR-ViT and MCR-ViT exhibit superior performance, which can be leveraged to build high-performance student behavior recognition systems in classrooms.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A cascaded residual vision transformer with wavelet transform and application in behavior recognition

  • Jing-Wei Liu,
  • Hao-Tian Ren,
  • Yu-Ran Du,
  • Jia-Ming Chen,
  • Jing Zhang

摘要

Objective

Convolutional Neural Networks (CNNs) have become essential tools for classroom student behavior recognition but lack the capability of global information capturing. In recent years, Vision Transformer (ViT) has demonstrated strong global modeling capabilities, which can be employed to strengthen the multi-level spatial information representation ability of classroom student behavior recognition models.

Methods

First, Cascaded Residual Vision Transformer (CR-ViT) model was proposed. The outputs of residual convolutional layers were integrated into multiple ViT modules to learn both shallow and deep feature representations for multi-level spatial information extraction, followed with the LSTM network for further capturing the global dependency between the sequences of ViT modules. Second, Cascaded Residual Vision Transformer with Morlet wavelet (MCR-ViT) was proposed. Based on CR-ViT, morlet wavelet transform activation layers were employed to improve the sensibility for variations of edges in feature maps.

Results

The proposed methods were validated on our collected dataset named Student Behavior in Classroom (SBIC), as well as the publicly available dataset of Student Classroom Behavior (SCB). The CR-ViT model and the MCR-ViT model improved the accuracy on SBIC by 6.10% and 7.32%, and on SCB by 0.92% and 1.14%. The MCR-ViT with second-order derivative of Morlet wavelet achieved the highest accuracy improvement compared to its variants based on other wavelet.

Conclusion and significance

both CR-ViT and MCR-ViT exhibit superior performance, which can be leveraged to build high-performance student behavior recognition systems in classrooms.