Scene segmentation plays a crucial role in self-environment perception for unmanned platforms in the Industrial Internet of Things (IIoT). However, while multi-modality semantic segmentation methods enhance performance, it is difficult to run in real-time on edge computing devices. Single-modality segmentation struggles to break through the bottleneck of performance and generalization due to its limited data form. To address this challenge, this paper proposes a hybrid-modality knowledge distillation method, 2T-1S (Two-Teacher-One-Student), to balance the trade-off between accuracy and inference speed. Specifically, we design a lightweight RGB segmentation encoder containing a Depth Feature Generation Module (DFGM) to extract simulated multi-dimensional depth feature from images through a feature generator. These features, together with the depth feature encoded by the teacher network, are used for soft-label distillation. Furthermore, we develop an efficient Fusion-Kan with only 0.45M parameters which directly acts on features from two different modalities to achieve end-to-end feature fusion. And then, we propose a cross-layer fusion decoder to enhance boundary segmentation. Finally, we use a complex RGB-D segmentation network as the teacher model. Hybrid distillation losses are applied to achieve the detailed semantic guidance from the multi-modal to the single-modal. Experimental results on multiple datasets demonstrate the effectiveness of the proposed 2T-1S, achieving superior performance compared to the state-of-the-art.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

2T-1S: A Lightweight Scene Segmentation Framework for IoT Devices Based on Hybrid-Modal Knowledge Distillation

  • Yaohua Liu,
  • Peng Qi,
  • Ruipeng Gao,
  • Dan Tao

摘要

Scene segmentation plays a crucial role in self-environment perception for unmanned platforms in the Industrial Internet of Things (IIoT). However, while multi-modality semantic segmentation methods enhance performance, it is difficult to run in real-time on edge computing devices. Single-modality segmentation struggles to break through the bottleneck of performance and generalization due to its limited data form. To address this challenge, this paper proposes a hybrid-modality knowledge distillation method, 2T-1S (Two-Teacher-One-Student), to balance the trade-off between accuracy and inference speed. Specifically, we design a lightweight RGB segmentation encoder containing a Depth Feature Generation Module (DFGM) to extract simulated multi-dimensional depth feature from images through a feature generator. These features, together with the depth feature encoded by the teacher network, are used for soft-label distillation. Furthermore, we develop an efficient Fusion-Kan with only 0.45M parameters which directly acts on features from two different modalities to achieve end-to-end feature fusion. And then, we propose a cross-layer fusion decoder to enhance boundary segmentation. Finally, we use a complex RGB-D segmentation network as the teacher model. Hybrid distillation losses are applied to achieve the detailed semantic guidance from the multi-modal to the single-modal. Experimental results on multiple datasets demonstrate the effectiveness of the proposed 2T-1S, achieving superior performance compared to the state-of-the-art.