Large language models (LLMs) increasingly drive intelligent services within automotive edge computing. However, deploying these models efficiently remains challenging due to diverse hardware setups and limited computational resources typical of automotive edge environments. Existing deployment strategies often disregard hardware diversity, resulting in suboptimal resource use and compromised performance, particularly under peak inference workloads. Consequently, computing elements like CPUs and integrated GPUs are frequently idle, with tasks excessively dependent on discrete GPUs. To address this, we propose a dynamic inference partitioning strategy named Hardware-Aware Dynamic Scheduling (ACL), tailored specifically for automotive edge computing. Our approach leverages the inherent distinction between initial token-processing phases (prefill) and subsequent token generation phases (decode) within LLM inference. By adaptively distributing these phases across heterogeneous hardware units, ACL maximizes resource utilization and balances workloads effectively. Empirical evaluations indicate that ACL significantly enhances inference performance. Furthermore, our framework demonstrates robust efficiency improvements consistently across various LLM architectures, highlighting its adaptability and effectiveness in heterogeneous automotive computing scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ACL: Adaptive Chunking of Large Language Models for Efficient Inference on Automotive Edge Devices

  • Yufei Lin,
  • Tianxiang Xu,
  • Chengwei Ye,
  • Huanzhen Zhang,
  • Kangsheng Wang

摘要

Large language models (LLMs) increasingly drive intelligent services within automotive edge computing. However, deploying these models efficiently remains challenging due to diverse hardware setups and limited computational resources typical of automotive edge environments. Existing deployment strategies often disregard hardware diversity, resulting in suboptimal resource use and compromised performance, particularly under peak inference workloads. Consequently, computing elements like CPUs and integrated GPUs are frequently idle, with tasks excessively dependent on discrete GPUs. To address this, we propose a dynamic inference partitioning strategy named Hardware-Aware Dynamic Scheduling (ACL), tailored specifically for automotive edge computing. Our approach leverages the inherent distinction between initial token-processing phases (prefill) and subsequent token generation phases (decode) within LLM inference. By adaptively distributing these phases across heterogeneous hardware units, ACL maximizes resource utilization and balances workloads effectively. Empirical evaluations indicate that ACL significantly enhances inference performance. Furthermore, our framework demonstrates robust efficiency improvements consistently across various LLM architectures, highlighting its adaptability and effectiveness in heterogeneous automotive computing scenarios.