Scholarlink

<p>Combining the complementary benefits of frames and events has become a widely adopted strategy for object detection in challenging scenarios, such as low-light conditions and high-speed motion blur. However, most multimodal object detection methods employ two independent Artificial Neural Network (ANN) branches to process frames and events, which limits cross-modality information interaction between the two visual streams and poses challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using asynchronous frames and events. Technically, we first present a semantic-enhanced self-attention mechanism for the Transformer architecture that explicitly models the relative semantic relationships between image encoding tokens to strengthen the correlation between them. Then, we design a Spiking Neural Network (SNN) branch, namely Spiking Swin Transformer, to extract temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks to align the asynchronous modalities and achieve cross-modality information interaction. The results show that our HDI-Former outperforms state-of-the-art methods and our baselines by a large margin. Our SNN branch achieves comparable performance to the ANN with the same architecture while consuming 10.57<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\times \)</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> and 11.46<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\times \)</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> less energy on the DSEC-Detection and PKU-DAVIS-SOD datasets, respectively. Our code is available at <a href="https://github.com/dianzl/hdiformer">https://github.com/dianzl/hdiformer</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Rethinking Neuromorphic Object Detection with Hybrid Dynamic Interaction Transformers

  • 期刊论文

  • 发表日期：

    2026年7月9日

查看全文
  • Dianze Li,
  • Jianing Li,
  • Xu Liu,
  • Zhaokun Zhou,
  • Xiaopeng Fan,
  • Yonghong Tian

摘要

Combining the complementary benefits of frames and events has become a widely adopted strategy for object detection in challenging scenarios, such as low-light conditions and high-speed motion blur. However, most multimodal object detection methods employ two independent Artificial Neural Network (ANN) branches to process frames and events, which limits cross-modality information interaction between the two visual streams and poses challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using asynchronous frames and events. Technically, we first present a semantic-enhanced self-attention mechanism for the Transformer architecture that explicitly models the relative semantic relationships between image encoding tokens to strengthen the correlation between them. Then, we design a Spiking Neural Network (SNN) branch, namely Spiking Swin Transformer, to extract temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks to align the asynchronous modalities and achieve cross-modality information interaction. The results show that our HDI-Former outperforms state-of-the-art methods and our baselines by a large margin. Our SNN branch achieves comparable performance to the ANN with the same architecture while consuming 10.57 \(\times \) × and 11.46 \(\times \) × less energy on the DSEC-Detection and PKU-DAVIS-SOD datasets, respectively. Our code is available at https://github.com/dianzl/hdiformer.