Night time traffic monitoring is hindered by low light, motion blur, and occlusions, restricting the performance of vision-based approaches. This article introduces a multimodal CNN-RNN model that combines infrared video and stereo audio for accurate vehicle detection and classification under low-light environments. Leveraging complementary acoustic-visual cues, the approach enhances detection precision for cars, buses, trucks, and motorbikes, including off-screen and partially occluded instances. When tested on actual night-time data, the new approach makes considerable gains over baseline performance. Box-wise IoU is 0.415, a relative increase of 18.2%, and point-wise IoU by 26.5%. Peak AUC scores reached 98.67% (validation) and 96.86% (test), demonstrating high class-separability and generalization. These results show that multimodal learning is indeed powerful for accurate traffic monitoring in difficult urban settings.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Audio-Visual Approach to Vehicle Detection and Classification

  • Sangita Jaybhaye,
  • Deepak Mane,
  • Yogesh Lale,
  • Apurva Kota,
  • Parth Kulkarni,
  • Tanvi Diwnale

摘要

Night time traffic monitoring is hindered by low light, motion blur, and occlusions, restricting the performance of vision-based approaches. This article introduces a multimodal CNN-RNN model that combines infrared video and stereo audio for accurate vehicle detection and classification under low-light environments. Leveraging complementary acoustic-visual cues, the approach enhances detection precision for cars, buses, trucks, and motorbikes, including off-screen and partially occluded instances. When tested on actual night-time data, the new approach makes considerable gains over baseline performance. Box-wise IoU is 0.415, a relative increase of 18.2%, and point-wise IoU by 26.5%. Peak AUC scores reached 98.67% (validation) and 96.86% (test), demonstrating high class-separability and generalization. These results show that multimodal learning is indeed powerful for accurate traffic monitoring in difficult urban settings.