Edge AI demands the deployment of complex neural networks under strict constraints on compute, memory, and power particularly in real-time applications such as autonomous vehicles, smart surveillance, and industrial inspection. We present a workload-aware partitioning strategy for neural networks on the u.RECS platform, a modular edge microserver integrating an NVIDIA Jetson Orin NX GPU and Hailo-8 AI accelerators in a compact Mini-ITX form factor. Using YOLOv7 as a case study, we demonstrate how model layers can be distributed across heterogeneous accelerators to exploit their complementary strengths, high-throughput parallelism on the GPU and energy-efficient inference on the Hailo-8. Our method minimizes inter-device communication by selecting a single, low-bandwidth split point and supports fully local inference. Experimental results show up to 1.7 \(\times \) performance gains and energy efficiency improvements over single-accelerator baselines, reaching 95 FPS at 3.3 FPS/Watt. These results demonstrate practical benefits of heterogeneous partitioning for far-edge scenarios where low latency, low power, and high throughput are essential.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Optimizing AI on the Edge: Partitioning Neural Networks Across Heterogeneous Accelerators

  • Kevin Mika,
  • Nils Kucza,
  • Florian Porrmann,
  • Jens Hagemeyer

摘要

Edge AI demands the deployment of complex neural networks under strict constraints on compute, memory, and power particularly in real-time applications such as autonomous vehicles, smart surveillance, and industrial inspection. We present a workload-aware partitioning strategy for neural networks on the u.RECS platform, a modular edge microserver integrating an NVIDIA Jetson Orin NX GPU and Hailo-8 AI accelerators in a compact Mini-ITX form factor. Using YOLOv7 as a case study, we demonstrate how model layers can be distributed across heterogeneous accelerators to exploit their complementary strengths, high-throughput parallelism on the GPU and energy-efficient inference on the Hailo-8. Our method minimizes inter-device communication by selecting a single, low-bandwidth split point and supports fully local inference. Experimental results show up to 1.7 \(\times \) performance gains and energy efficiency improvements over single-accelerator baselines, reaching 95 FPS at 3.3 FPS/Watt. These results demonstrate practical benefits of heterogeneous partitioning for far-edge scenarios where low latency, low power, and high throughput are essential.