Scholarlink

<p>Deploying large language models (LLMs) on resource-constrained edge devices remains a significant systems challenge due to limited CPU capacity, restricted memory, absence of hardware accelerators, and the need for low-latency, privacy-preserving operation without cloud connectivity. Existing approaches primarily emphasize model compression, energy profiling, or hybrid cloud–edge architectures, but lack systematic evaluation of end-to-end LLM behavior when exposed as concurrent, web-accessible services on single-device edge platforms. In particular, the interaction between quantized LLM inference, asynchronous web serving, and request-level concurrency remains poorly understood. This paper presents <i>SanicLLMPiAB</i>, a fully localized and internet-independent framework for evaluating quantized LLM inference on a Raspberry&#xa0;Pi 4B using an asynchronous Sanic web server, the Ollama local LLM runtime, and Apache Bench–driven stress testing. Unlike prior model-centric studies, the proposed framework enables fine-grained, per-request observability by capturing token-level execution metrics, end-to-end latency, CPU utilization, memory stability, and accuracy under controlled concurrency. All inference is executed locally, ensuring data privacy and eliminating network variability. Experimental evaluation of four quantized LLMs (Qwen&#xa0;0.5B, TinyLLaMA, TinyDolphin, and StableLM2) demonstrates that Qwen&#xa0;0.5B achieves the best balance of scalability and robustness, sustaining stable throughput (11–12 tokens/s), sub-second end-to-end latency at higher concurrency, and 100% response accuracy. In contrast, larger models exhibit severe latency inflation, instability, and execution failures under moderate load. Statistical analyses (ANOVA, MANOVA, CCA, and hierarchical clustering) confirm concurrency as the dominant factor shaping edge inference behavior. These results establish SanicLLMPiAB as a practical, reproducible foundation for secure and scalable LLM deployment on edge devices. The code is hosted on <a href="https://github.com/ParthaPRay/Sanic-Ollama-RaspberryPi-API">https://github.com/ParthaPRay/Sanic-Ollama-RaspberryPi-API</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SanicLLMPiAB: Sanic Web Framework on Raspberry Pi for Localized LLMs for Apache Benchmark Analysis

  • 期刊论文

  • 发表日期：

    2026年6月15日

查看全文
  • Partha Pratim Ray,
  • Mohan Pratap Pradhan

摘要

Deploying large language models (LLMs) on resource-constrained edge devices remains a significant systems challenge due to limited CPU capacity, restricted memory, absence of hardware accelerators, and the need for low-latency, privacy-preserving operation without cloud connectivity. Existing approaches primarily emphasize model compression, energy profiling, or hybrid cloud–edge architectures, but lack systematic evaluation of end-to-end LLM behavior when exposed as concurrent, web-accessible services on single-device edge platforms. In particular, the interaction between quantized LLM inference, asynchronous web serving, and request-level concurrency remains poorly understood. This paper presents SanicLLMPiAB, a fully localized and internet-independent framework for evaluating quantized LLM inference on a Raspberry Pi 4B using an asynchronous Sanic web server, the Ollama local LLM runtime, and Apache Bench–driven stress testing. Unlike prior model-centric studies, the proposed framework enables fine-grained, per-request observability by capturing token-level execution metrics, end-to-end latency, CPU utilization, memory stability, and accuracy under controlled concurrency. All inference is executed locally, ensuring data privacy and eliminating network variability. Experimental evaluation of four quantized LLMs (Qwen 0.5B, TinyLLaMA, TinyDolphin, and StableLM2) demonstrates that Qwen 0.5B achieves the best balance of scalability and robustness, sustaining stable throughput (11–12 tokens/s), sub-second end-to-end latency at higher concurrency, and 100% response accuracy. In contrast, larger models exhibit severe latency inflation, instability, and execution failures under moderate load. Statistical analyses (ANOVA, MANOVA, CCA, and hierarchical clustering) confirm concurrency as the dominant factor shaping edge inference behavior. These results establish SanicLLMPiAB as a practical, reproducible foundation for secure and scalable LLM deployment on edge devices. The code is hosted on https://github.com/ParthaPRay/Sanic-Ollama-RaspberryPi-API.