With continuous development of computer vision technology, object detection models based on single-modal image (e.g., visible light images) have become increasingly insufficient to meet the demand of complex environments. Particularly, in low-light conditions, such as nighttime, extreme weather, etc., the quality of visible light images deteriorates significantly, leading to notable performance degradation of traditional object detection methods. In contrast, infrared images demonstrate superior robustness in such challenging conditions. Consequently, integrating visible and infrared images (i.e., multimodal fusion) for object detection has emerged as a critical approach to enhance detection performance. In this study, we propose a dual-modal fusion object detection model based on the Faster R-CNN, by fusing visible and infrared images. We investigate different fusion strategies, including input-level and feature-level fusion, and employ a dual-branch backbone network (ResNet50 with Feature Pyramid Network(FPN)) to process the two modalities, effectively leveraging their complementary strengths. Experimental results demonstrate that feature-level fusion significantly improves detection accuracy under low-light and extreme weather conditions. Meanwhile, the propose method increases mAP50 from 96.60% (single-modal) to 97.90% (dual-modal) and improves mAP50:95 from 54.01% (single-modal) to 64.80% (dual-modal) on the LLVIP dataset, validating the effectiveness of our approach.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-modal Object Detection with Improved Faster R-CNN

  • Jiabao Qian,
  • Jiyuan Liu,
  • Rusheng Ju

摘要

With continuous development of computer vision technology, object detection models based on single-modal image (e.g., visible light images) have become increasingly insufficient to meet the demand of complex environments. Particularly, in low-light conditions, such as nighttime, extreme weather, etc., the quality of visible light images deteriorates significantly, leading to notable performance degradation of traditional object detection methods. In contrast, infrared images demonstrate superior robustness in such challenging conditions. Consequently, integrating visible and infrared images (i.e., multimodal fusion) for object detection has emerged as a critical approach to enhance detection performance. In this study, we propose a dual-modal fusion object detection model based on the Faster R-CNN, by fusing visible and infrared images. We investigate different fusion strategies, including input-level and feature-level fusion, and employ a dual-branch backbone network (ResNet50 with Feature Pyramid Network(FPN)) to process the two modalities, effectively leveraging their complementary strengths. Experimental results demonstrate that feature-level fusion significantly improves detection accuracy under low-light and extreme weather conditions. Meanwhile, the propose method increases mAP50 from 96.60% (single-modal) to 97.90% (dual-modal) and improves mAP50:95 from 54.01% (single-modal) to 64.80% (dual-modal) on the LLVIP dataset, validating the effectiveness of our approach.