As one of the most fundamental communication paradigms in distributed systems, Allreduce is critical for efficient data synchronization in large-scale machine learning and high-performance computing. While traditional software-based Allreduce implementations exhibit inherent limitations such as high latency and low throughput, in-network aggregation technology mitigates these issues by offloading computation to programmable network devices. However, existing deployment approaches overly focus on host-side resource allocation while neglecting network link load balancing, ultimately restricting system throughput. To address this challenge, we propose the Dynamic job Placement Strategy (DPS), which leverages a tree-topology network with in-network aggregation support to jointly optimize computational resource allocation and network load balancing, thereby achieving high-throughput, low-latency distributed job placement. Extensive experimental results demonstrate DPS’s superior performance, showing a 3 \(\times \) throughput enhancement for individual Allreduce operations and a 19.3% reduction in average completion time for concurrent Allreduce workloads compared to state-of-the-art alternatives.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DPS: A Congestion-Aware Allreduce Job Placement for In-Network Aggregation

  • Yanrong Hu,
  • Guannan Zhang,
  • Dezun Dong,
  • Zihao Wei,
  • Zhen Ruan

摘要

As one of the most fundamental communication paradigms in distributed systems, Allreduce is critical for efficient data synchronization in large-scale machine learning and high-performance computing. While traditional software-based Allreduce implementations exhibit inherent limitations such as high latency and low throughput, in-network aggregation technology mitigates these issues by offloading computation to programmable network devices. However, existing deployment approaches overly focus on host-side resource allocation while neglecting network link load balancing, ultimately restricting system throughput. To address this challenge, we propose the Dynamic job Placement Strategy (DPS), which leverages a tree-topology network with in-network aggregation support to jointly optimize computational resource allocation and network load balancing, thereby achieving high-throughput, low-latency distributed job placement. Extensive experimental results demonstrate DPS’s superior performance, showing a 3 \(\times \) throughput enhancement for individual Allreduce operations and a 19.3% reduction in average completion time for concurrent Allreduce workloads compared to state-of-the-art alternatives.