The emergence of trillion-parameter large language models(LLMs) like GPT-4 and PaLM-2 has fundamentally transformed artificial general intelligence capabilities. However, their inefficient synchronization strategy and suboptimal network resource utilization together limit the system’s scalability and training efficiency. This survey provides a systematic categorization of communication optimization techniques. It introduces a hierarchical framework that differentiates between logical strategy optimization and physical resource orchestration. Strategy optimizations focus on adaptive synchronization and gradient compression to reduce communication volume. Resource optimizations improve task scheduling, network topology, and hardware adaptation to boost physical efficiency. This work classifies current optimization strategies for large-scale distributed training systems. Further, it discusses potential research directions for improving communication efficiency in next-generation LLM distributed training systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Communication Optimization for Large Language Model Distributed Training: A Systematic Survey

  • Sheng Li,
  • Jun Zhu,
  • Yuanhao He,
  • Hanguang Luo,
  • Tao Zou,
  • Geyang Xiao,
  • Donghui Lu

摘要

The emergence of trillion-parameter large language models(LLMs) like GPT-4 and PaLM-2 has fundamentally transformed artificial general intelligence capabilities. However, their inefficient synchronization strategy and suboptimal network resource utilization together limit the system’s scalability and training efficiency. This survey provides a systematic categorization of communication optimization techniques. It introduces a hierarchical framework that differentiates between logical strategy optimization and physical resource orchestration. Strategy optimizations focus on adaptive synchronization and gradient compression to reduce communication volume. Resource optimizations improve task scheduling, network topology, and hardware adaptation to boost physical efficiency. This work classifies current optimization strategies for large-scale distributed training systems. Further, it discusses potential research directions for improving communication efficiency in next-generation LLM distributed training systems.