Scholarlink

The rapid advancement of Large Language Models (LLMs) has provided new opportunities for edge applications. Embedded FPGA platforms are well-suited for deploying LLMs at the edge due to their hardware programmability and high energy efficiency. However, the limited hardware resources of embedded systems pose significant challenges to comprehensive acceleration. As not all operators can be offloaded to programmable logic (PL), the remaining operations must execute on the processing system (PS), creating potential performance bottlenecks and degrading inference efficiency. In this paper, we introduce Lembda, a collaborative optimization framework that harnesses the computational capability of both PL and PS for efficient embedded LLM inference. On the PL side, we employ W4A8 quantization and implement high-throughput GEMM/GEMV kernels. On the PS side, we optimize high-precision operations by exploiting model sparsity in attention layers and approximate nonlinear functions via lightweight polynomial fitting. Moreover, we carefully orchestrate PL/PS operations to exploit operation-level parallelism and further enhance performance. Evaluations on the AMD Kria KV260 platform demonstrates that Lembda delivers \(187.9195\ \mathrm {tok/s}\) for prefilling and \(9.7857\ \mathrm {tok/s}\) for decoding on the Qwen2.5-0.5B-Instruct model, achieving \(65.9\times \) / \(3.8\times \) speedup compared to the baseline method with negligible accuracy loss.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Lembda: Optimizing LLM Inference on Embedded Platforms via CPU/FPGA Co-processing

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Jinwei Zhou,
  • Chenhao Xue,
  • Xiping Dong,
  • Yi Ren,
  • Jiaxing Zhang,
  • Guangyu Sun,
  • Xinnan Lin

摘要

The rapid advancement of Large Language Models (LLMs) has provided new opportunities for edge applications. Embedded FPGA platforms are well-suited for deploying LLMs at the edge due to their hardware programmability and high energy efficiency. However, the limited hardware resources of embedded systems pose significant challenges to comprehensive acceleration. As not all operators can be offloaded to programmable logic (PL), the remaining operations must execute on the processing system (PS), creating potential performance bottlenecks and degrading inference efficiency. In this paper, we introduce Lembda, a collaborative optimization framework that harnesses the computational capability of both PL and PS for efficient embedded LLM inference. On the PL side, we employ W4A8 quantization and implement high-throughput GEMM/GEMV kernels. On the PS side, we optimize high-precision operations by exploiting model sparsity in attention layers and approximate nonlinear functions via lightweight polynomial fitting. Moreover, we carefully orchestrate PL/PS operations to exploit operation-level parallelism and further enhance performance. Evaluations on the AMD Kria KV260 platform demonstrates that Lembda delivers \(187.9195\ \mathrm {tok/s}\) for prefilling and \(9.7857\ \mathrm {tok/s}\) for decoding on the Qwen2.5-0.5B-Instruct model, achieving \(65.9\times \) / \(3.8\times \) speedup compared to the baseline method with negligible accuracy loss.