Large language model(LLM) training relies on multiple parallel strategies, where high-bandwidth domains (HBDs) play a key role in enabling efficient communication between NPUs. Compared to intra-HBD communication, cross-HBD communication is significantly slower and remains unavoidable due to limited HBD sizes and different model partitioning strategies. Hierarchical collectives can reduce cross-HBD communication overhead, but their effectiveness is hindered by imbalanced parallel group distribution across HBDs, different communication costs with different distribution patterns, and interdependencies among parallelism strategies. To address these challenges, we propose HBD-CE, a model placement scheme designed for communication-efficient cross-HBD LLM training in HBD clusters, which optimizes model placement to enhance cross-HBD communication efficiency. We formulate the model placement problem as a mixed-integer nonlinear programming (MINLP) problem. By leveraging the properties of hierarchical collectives, we transform the MINLP into a series of 0–1 integer linear programming (ILP) problems and develop an exact algorithm to solve them. Experiment results demonstrate that HBD-CE can reduce per-iteration communication time by 5%–77.3% across diverse topologies and model configurations.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

HBD-CE: Efficient Cross-HBD Communication for LLM Training in High-Bandwidth Domain Cluster via Hierarchical Collectives

  • Huihuang Qin,
  • Shuangwu Chen,
  • Zijian Wen,
  • Zian Wang,
  • Ziyang Zou,
  • Tao Zhang,
  • Xiaobin Tan,
  • Jian Yang

摘要

Large language model(LLM) training relies on multiple parallel strategies, where high-bandwidth domains (HBDs) play a key role in enabling efficient communication between NPUs. Compared to intra-HBD communication, cross-HBD communication is significantly slower and remains unavoidable due to limited HBD sizes and different model partitioning strategies. Hierarchical collectives can reduce cross-HBD communication overhead, but their effectiveness is hindered by imbalanced parallel group distribution across HBDs, different communication costs with different distribution patterns, and interdependencies among parallelism strategies. To address these challenges, we propose HBD-CE, a model placement scheme designed for communication-efficient cross-HBD LLM training in HBD clusters, which optimizes model placement to enhance cross-HBD communication efficiency. We formulate the model placement problem as a mixed-integer nonlinear programming (MINLP) problem. By leveraging the properties of hierarchical collectives, we transform the MINLP into a series of 0–1 integer linear programming (ILP) problems and develop an exact algorithm to solve them. Experiment results demonstrate that HBD-CE can reduce per-iteration communication time by 5%–77.3% across diverse topologies and model configurations.