Vision transformers (ViTs) have become the go-to solution in many computer vision tasks due to their ability to capture long-range dependencies in data and, consequently, achieve remarkable performance in dense prediction tasks. However, their high computational and memory requirements make these architectures unsuitable for embedded and low-power devices. This work introduces a novel linear ViT architecture optimized for resource-constrained embedded hardware. Our model operates with less than 1MB of parameters and 400KB of RAM, relying only on operations compatible with most embedded runtimes. We propose a linear attention mechanism based solely on fully connected layers, enabling execution on platforms that do not support typical transformer operations. To ease deployment, we apply a Hardware-Aware Scaling (HAS) strategy that enables one-shot network scaling to meet specific hardware constraints, without requiring an expensive Network Architecture Search (NAS). Using just three hyperparameters, HAS supports independent control over model size, memory usage, and computational cost. Our network architecture shows better generalization performance and up to 80% lower latency compared to current alternatives for embedded devices for Imagenet-1K classification. When coupled with a DETR detection head, we achieve state-of-the-art mAP/latency tradeoffs and up to a \(90\%\) reduction in latency compared to existing approaches, thanks to better optimization for the lightweight CPUs and small accelerators typical of tiny, resource-constrained devices.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Efficient Attention on Microcontrollers via Linearized Transformer Blocks

  • Alberto Ancilotto,
  • Edoardo Castagnini,
  • Elisabetta Farella

摘要

Vision transformers (ViTs) have become the go-to solution in many computer vision tasks due to their ability to capture long-range dependencies in data and, consequently, achieve remarkable performance in dense prediction tasks. However, their high computational and memory requirements make these architectures unsuitable for embedded and low-power devices. This work introduces a novel linear ViT architecture optimized for resource-constrained embedded hardware. Our model operates with less than 1MB of parameters and 400KB of RAM, relying only on operations compatible with most embedded runtimes. We propose a linear attention mechanism based solely on fully connected layers, enabling execution on platforms that do not support typical transformer operations. To ease deployment, we apply a Hardware-Aware Scaling (HAS) strategy that enables one-shot network scaling to meet specific hardware constraints, without requiring an expensive Network Architecture Search (NAS). Using just three hyperparameters, HAS supports independent control over model size, memory usage, and computational cost. Our network architecture shows better generalization performance and up to 80% lower latency compared to current alternatives for embedded devices for Imagenet-1K classification. When coupled with a DETR detection head, we achieve state-of-the-art mAP/latency tradeoffs and up to a \(90\%\) reduction in latency compared to existing approaches, thanks to better optimization for the lightweight CPUs and small accelerators typical of tiny, resource-constrained devices.