Modern deep learning and HPC applications on distributed GPU clusters rely on collective communication primitives such as Broadcast, All-Reduce, and All-Gather for efficient data coordination across devices and nodes. The current OpenMP specification lacks native support for multi-device collectives within its target offloading model, leading developers to re-implement algorithms or depend on vendor-specific libraries. Recent OpenMP extensions aim to support distributed clusters, but naive collectives in such environments often suffer from performance issues due to network topology heterogeneity. In this paper, we present the OpenMP Collective Communication Library (OMPCCL), an extension to the OpenMP Target model that integrates portable collective primitives and device group semantics into the OpenMP interface. We prototype several collectives including Broadcast, All-Reduce, and Reduce-Scatter and evaluate performance on a 64-GPU cluster. Results show up to \(5\times \) speedup over naive methods, while maintaining OpenMP’s portability and simplicity. We also analyze the influence of network topology on collective performance and provide guidelines for efficient OpenMP collectives on GPU clusters.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

OMPCCL: Extending OpenMP with Portable Collective Operations for Multiple Devices

  • Jhonatan Cléto,
  • Rémy Neveu,
  • Rodrigo Ceccato,
  • Guilherme Valarini,
  • Jose M. Monsalve Diaz,
  • Hervé Yviquel

摘要

Modern deep learning and HPC applications on distributed GPU clusters rely on collective communication primitives such as Broadcast, All-Reduce, and All-Gather for efficient data coordination across devices and nodes. The current OpenMP specification lacks native support for multi-device collectives within its target offloading model, leading developers to re-implement algorithms or depend on vendor-specific libraries. Recent OpenMP extensions aim to support distributed clusters, but naive collectives in such environments often suffer from performance issues due to network topology heterogeneity. In this paper, we present the OpenMP Collective Communication Library (OMPCCL), an extension to the OpenMP Target model that integrates portable collective primitives and device group semantics into the OpenMP interface. We prototype several collectives including Broadcast, All-Reduce, and Reduce-Scatter and evaluate performance on a 64-GPU cluster. Results show up to \(5\times \) speedup over naive methods, while maintaining OpenMP’s portability and simplicity. We also analyze the influence of network topology on collective performance and provide guidelines for efficient OpenMP collectives on GPU clusters.