Modern supercomputers feature an ever-increasing degree of parallelism, especially in the number of cores per node. These high core counts are considered in our flexible implementation of persistent allreduce (an MPI-4 feature), which was implemented specifically with shared-memory communication in mind. At a high level, our algorithm consists of a reduce_scatter stage followed by an allgather stage, and allows for different factors (i.e., multi-radix) to be applied at each. Where barriers are required, they are integrated into the algorithm using counters to track progress. In order to accommodate the complexity of this approach, our implementation is split into a setup phase and an execution phase. The setup phase only occurs once for a given set of parameters, and is responsible for determining the algorithm that will be run each time the allreduce is called in the execution phase. Using these methods, we achieve speedups of half an order of magnitude compared to the blocking and persistent allreduce implementations of MPICH and OpenMPI, on a dual socket node with AMD EPYC processors and almost an order of magnitude on a four socket node with NVIDIA Grace processors. Our implementation also achieves good performance on multiple nodes.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Flexible Algorithms for Persistent MPI Allreduce Communication

  • Andreas Jocksch,
  • C. Nicole Avans,
  • Riley Shipley,
  • Anthony Skjellum

摘要

Modern supercomputers feature an ever-increasing degree of parallelism, especially in the number of cores per node. These high core counts are considered in our flexible implementation of persistent allreduce (an MPI-4 feature), which was implemented specifically with shared-memory communication in mind. At a high level, our algorithm consists of a reduce_scatter stage followed by an allgather stage, and allows for different factors (i.e., multi-radix) to be applied at each. Where barriers are required, they are integrated into the algorithm using counters to track progress. In order to accommodate the complexity of this approach, our implementation is split into a setup phase and an execution phase. The setup phase only occurs once for a given set of parameters, and is responsible for determining the algorithm that will be run each time the allreduce is called in the execution phase. Using these methods, we achieve speedups of half an order of magnitude compared to the blocking and persistent allreduce implementations of MPICH and OpenMPI, on a dual socket node with AMD EPYC processors and almost an order of magnitude on a four socket node with NVIDIA Grace processors. Our implementation also achieves good performance on multiple nodes.