Communication Optimization for Large Language Model Distributed Training: A Systematic Survey
摘要
The emergence of trillion-parameter large language models(LLMs) like GPT-4 and PaLM-2 has fundamentally transformed artificial general intelligence capabilities. However, their inefficient synchronization strategy and suboptimal network resource utilization together limit the system’s scalability and training efficiency. This survey provides a systematic categorization of communication optimization techniques. It introduces a hierarchical framework that differentiates between logical strategy optimization and physical resource orchestration. Strategy optimizations focus on adaptive synchronization and gradient compression to reduce communication volume. Resource optimizations improve task scheduling, network topology, and hardware adaptation to boost physical efficiency. This work classifies current optimization strategies for large-scale distributed training systems. Further, it discusses potential research directions for improving communication efficiency in next-generation LLM distributed training systems.