HBD-CE: Efficient Cross-HBD Communication for LLM Training in High-Bandwidth Domain Cluster via Hierarchical Collectives
摘要
Large language model(LLM) training relies on multiple parallel strategies, where high-bandwidth domains (HBDs) play a key role in enabling efficient communication between NPUs. Compared to intra-HBD communication, cross-HBD communication is significantly slower and remains unavoidable due to limited HBD sizes and different model partitioning strategies. Hierarchical collectives can reduce cross-HBD communication overhead, but their effectiveness is hindered by imbalanced parallel group distribution across HBDs, different communication costs with different distribution patterns, and interdependencies among parallelism strategies. To address these challenges, we propose HBD-CE, a model placement scheme designed for communication-efficient cross-HBD LLM training in HBD clusters, which optimizes model placement to enhance cross-HBD communication efficiency. We formulate the model placement problem as a mixed-integer nonlinear programming (MINLP) problem. By leveraging the properties of hierarchical collectives, we transform the MINLP into a series of 0–1 integer linear programming (ILP) problems and develop an exact algorithm to solve them. Experiment results demonstrate that HBD-CE can reduce per-iteration communication time by 5%–77.3% across diverse topologies and model configurations.