The deployment and inference of Deep Learning models are investigated in this article on seven different Edge Computing (EC) devices. This study addresses the features, performance and limitations of the NVIDIA Jetson Orin NX and Nano, Google Coral DevBoard and USB, Intel Neural Compute Stick 2, NXP i.MX8 Plus and Xilinx Zynq UltraScale+ MPSoC ZCU104 on Deep Learning inference. Fully Connected, Convolutional and Long Short-Term Memory (LSTM) neural networks are implemented to test these EC devices. The benchmarking focuses on the performance metrics: inference latency, increase in error metric, and power consumption. The results show considerable variability among devices, with the ZCU104 and Jetson Orin achieving the lowest latencies across most models without any increase in the error metric. At the same time, Coral devices exhibit increased latency and error for complex convolutional models. NVIDIA Jetson devices, ZCU104 and Neural Compute Stick 2 are the only devices that support LSTM inference. The study also highlights differences in power consumption, with USB accelerators being the most energy-efficient.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Deep Learning Inference on Edge: A Preliminary Device Comparison

  • Manuel L. González,
  • Jorge Ruiz,
  • Lidia Andrés,
  • Randy Lozada,
  • Erik S. Skibinsky,
  • Jorge Fernández,
  • Javier Sedano,
  • Ángel M. García-Vico

摘要

The deployment and inference of Deep Learning models are investigated in this article on seven different Edge Computing (EC) devices. This study addresses the features, performance and limitations of the NVIDIA Jetson Orin NX and Nano, Google Coral DevBoard and USB, Intel Neural Compute Stick 2, NXP i.MX8 Plus and Xilinx Zynq UltraScale+ MPSoC ZCU104 on Deep Learning inference. Fully Connected, Convolutional and Long Short-Term Memory (LSTM) neural networks are implemented to test these EC devices. The benchmarking focuses on the performance metrics: inference latency, increase in error metric, and power consumption. The results show considerable variability among devices, with the ZCU104 and Jetson Orin achieving the lowest latencies across most models without any increase in the error metric. At the same time, Coral devices exhibit increased latency and error for complex convolutional models. NVIDIA Jetson devices, ZCU104 and Neural Compute Stick 2 are the only devices that support LSTM inference. The study also highlights differences in power consumption, with USB accelerators being the most energy-efficient.