ACL: Adaptive Chunking of Large Language Models for Efficient Inference on Automotive Edge Devices
摘要
Large language models (LLMs) increasingly drive intelligent services within automotive edge computing. However, deploying these models efficiently remains challenging due to diverse hardware setups and limited computational resources typical of automotive edge environments. Existing deployment strategies often disregard hardware diversity, resulting in suboptimal resource use and compromised performance, particularly under peak inference workloads. Consequently, computing elements like CPUs and integrated GPUs are frequently idle, with tasks excessively dependent on discrete GPUs. To address this, we propose a dynamic inference partitioning strategy named Hardware-Aware Dynamic Scheduling (ACL), tailored specifically for automotive edge computing. Our approach leverages the inherent distinction between initial token-processing phases (prefill) and subsequent token generation phases (decode) within LLM inference. By adaptively distributing these phases across heterogeneous hardware units, ACL maximizes resource utilization and balances workloads effectively. Empirical evaluations indicate that ACL significantly enhances inference performance. Furthermore, our framework demonstrates robust efficiency improvements consistently across various LLM architectures, highlighting its adaptability and effectiveness in heterogeneous automotive computing scenarios.