In automatic driving technology, 3D object detection by multi-sensor fusion faces the serious challenge of sensor performance degradation under bad weather. Existing methods rely on static feature splicing strategy, which is prone to feature misalignment and information loss under complex weather conditions such as rain, snow and fog, and lacks a dynamic adjustment mechanism for modal contribution, resulting in information loss and increased noise interference. To address this problem, this paper proposes an adaptive multimodal fusion framework, AWAD-Fusion, which realizes bidirectional enhancement and dynamic weight allocation of image and LiDAR features through the cross-modal feature interaction module (CFIM) and adaptive gated dynamic fusion module (AGDF) in bird’s-eye view (BEV) space. CFIM utilizes the attention mechanism to establish the cross-modal geometric alignment relationship, which effectively compensates for the feature degradation and deviation caused by weather interference; AGDF evaluates the modal reliability in real time through the gated weights, and dynamically suppresses the interference of noisy modes. Experiments show that on the nuScenes dataset, the method achieves 69.7% mAP and 72.3% NDS in conventional scenes, which are 0.8% and 1.1% higher than the baseline, respectively; and in the comprehensive adverse weather test, the mAP and NDS reach 43.1% and 47.5%, which are 19.7% and 13.6% higher than the baseline, providing innovative and robust sensing methods for the adverse weather environments. This provides an innovative solution for robust perception in adverse weather environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AWAD-Fusion: Dynamic Multi-sensor Fusion Framework for Robust 3D Object Detection in Adverse Weather

  • Xinyu Yang,
  • Ling Wang

摘要

In automatic driving technology, 3D object detection by multi-sensor fusion faces the serious challenge of sensor performance degradation under bad weather. Existing methods rely on static feature splicing strategy, which is prone to feature misalignment and information loss under complex weather conditions such as rain, snow and fog, and lacks a dynamic adjustment mechanism for modal contribution, resulting in information loss and increased noise interference. To address this problem, this paper proposes an adaptive multimodal fusion framework, AWAD-Fusion, which realizes bidirectional enhancement and dynamic weight allocation of image and LiDAR features through the cross-modal feature interaction module (CFIM) and adaptive gated dynamic fusion module (AGDF) in bird’s-eye view (BEV) space. CFIM utilizes the attention mechanism to establish the cross-modal geometric alignment relationship, which effectively compensates for the feature degradation and deviation caused by weather interference; AGDF evaluates the modal reliability in real time through the gated weights, and dynamically suppresses the interference of noisy modes. Experiments show that on the nuScenes dataset, the method achieves 69.7% mAP and 72.3% NDS in conventional scenes, which are 0.8% and 1.1% higher than the baseline, respectively; and in the comprehensive adverse weather test, the mAP and NDS reach 43.1% and 47.5%, which are 19.7% and 13.6% higher than the baseline, providing innovative and robust sensing methods for the adverse weather environments. This provides an innovative solution for robust perception in adverse weather environments.