Neural networks based on State Space Models (SSMs) have shown good performance on long sequence modeling tasks, such as raw audio classification. So far, their continuous-time parameter representation has not been used for analog neural network computing. We propose AnalogSSM, a diagonal and real-valued SSM architecture that can be converted into a purely analog electric circuit representation consisting of adder/subtraction, low-pass, and rectifier operational amplifier circuits. Targeting hotword detection based on the Google Speech Commands dataset, we evaluate three model configurations ranging from 0.15k – 1.3k parameters. Achieving, on average, over ten individual hotwords, an accuracy range of 84.5% – 90.8% with discrete models in PyTorch. The synthesized electric circuits are simulated and evaluated with LTspice. On average, we observe accuracy drops of 2.9pp with the continuous-time analog circuits only consisting of 70 – 238 operational amplifiers.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Modeling Diagonal State Space Models as Electric Circuits for Analog Neural Network Inference

  • Matthias Bittner,
  • Daniel Schnöll,
  • Fabian Seiler,
  • Matthias Wess,
  • Axel Jantsch

摘要

Neural networks based on State Space Models (SSMs) have shown good performance on long sequence modeling tasks, such as raw audio classification. So far, their continuous-time parameter representation has not been used for analog neural network computing. We propose AnalogSSM, a diagonal and real-valued SSM architecture that can be converted into a purely analog electric circuit representation consisting of adder/subtraction, low-pass, and rectifier operational amplifier circuits. Targeting hotword detection based on the Google Speech Commands dataset, we evaluate three model configurations ranging from 0.15k – 1.3k parameters. Achieving, on average, over ten individual hotwords, an accuracy range of 84.5% – 90.8% with discrete models in PyTorch. The synthesized electric circuits are simulated and evaluated with LTspice. On average, we observe accuracy drops of 2.9pp with the continuous-time analog circuits only consisting of 70 – 238 operational amplifiers.