Decoding and Deployment
摘要
After training is complete, large language models (LLMs) can be deployed in real-world applications. In the autoregressive architecture, the model generates output content token by token based on the input, often referred to as the prompt text—a process known as decoding. This chapter begins by introducing common decoding strategies in the autoregressive framework, focusing on two fundamental approaches: greedy search and random sampling. We then explore enhanced versions of these strategies, including beam search, length penalty, and repetition penalty for greedy search, as well as temperature sampling, top-k sampling, and top-p sampling for random sampling. Next, we address the challenge of low decoding efficiency in autoregressive generation by providing a quantitative analysis of model inference efficiency. Specifically, we break down the process into individual phases—prefill and decoding—and present methods for calculating memory costs for each phase. To further improve efficiency, we introduce optimization techniques at both the system and algorithm levels. For system-level optimization, we briefly discuss three prominent methods: FlashAttention, PagedAttention, and batch management. For algorithm-level optimization, we cover speculative decoding, cascade decoding, non-autoregressive decoding, and early exit mechanisms. Finally, we introduce model quantization, a widely used compression technique for deploying LLMs in resource-constrained environments. We also briefly touch on other compression methods, including model distillation and pruning. This chapter aims to provide a comprehensive overview of decoding strategies, efficiency optimization, and deployment techniques, equipping practitioners with the techniques needed to effectively utilize LLMs in real-world scenarios.