<p>Accurate 3D object detection is fundamental for autonomous driving systems, but existing state-of-the-art methods are yet to overcome challenges in terms of achieving consistent and robust results. The two main difficulties in multimodal perception are the sparse nature of 4D mm-wave radar point clouds, and the modality gap between radar and camera features for fusion in a common representation space. Such problems can adversely affect the accuracy of detection, especially for small and far away objects. To cope with these problems, this paper presents an Adaptive Feature Alignment Cross-modal Radar-camera Fusion Framework (CRFF-Net) based on radar point cloud super-resolution. A spatial module based on Graph Convolutional Network (GCN) is proposed to enhance the spatial representation of radar point clouds by producing denser and geometrically consistent ones. Moreover, a deformable cross-attention mechanism is introduced to match the features of radar and image in the Bird's Eye View (BEV) space adaptively, which can alleviate the loss of information that occurs in traditional fusion methods. The result indicates that CRFF-Net has a mean Average Precision (mAP) of 57.43%, which is superior to the strong radar-camera fusion basis, LXL, by 1.12%, and the result is verified by extensive experiments on the View-of-Delft (VoD) data set. The proposed method also demonstrates its robustness and maintains high detection performance under noisy conditions compared to the existing methods. The findings suggest that the adaptive cross-modal fusion of the super-resolution radar data and the video image can effectively and reliably detect the 3D shape of the object in complex driving environments.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CRFF-Net: cross-modal fusion of 4D mmWave radar and camera with super-resolution for 3D object detection

  • Abeer Aljohani

摘要

Accurate 3D object detection is fundamental for autonomous driving systems, but existing state-of-the-art methods are yet to overcome challenges in terms of achieving consistent and robust results. The two main difficulties in multimodal perception are the sparse nature of 4D mm-wave radar point clouds, and the modality gap between radar and camera features for fusion in a common representation space. Such problems can adversely affect the accuracy of detection, especially for small and far away objects. To cope with these problems, this paper presents an Adaptive Feature Alignment Cross-modal Radar-camera Fusion Framework (CRFF-Net) based on radar point cloud super-resolution. A spatial module based on Graph Convolutional Network (GCN) is proposed to enhance the spatial representation of radar point clouds by producing denser and geometrically consistent ones. Moreover, a deformable cross-attention mechanism is introduced to match the features of radar and image in the Bird's Eye View (BEV) space adaptively, which can alleviate the loss of information that occurs in traditional fusion methods. The result indicates that CRFF-Net has a mean Average Precision (mAP) of 57.43%, which is superior to the strong radar-camera fusion basis, LXL, by 1.12%, and the result is verified by extensive experiments on the View-of-Delft (VoD) data set. The proposed method also demonstrates its robustness and maintains high detection performance under noisy conditions compared to the existing methods. The findings suggest that the adaptive cross-modal fusion of the super-resolution radar data and the video image can effectively and reliably detect the 3D shape of the object in complex driving environments.