Scalable OpenMP Remote Offloading via Asynchronous MPI and Coroutine-Driven Communication
摘要
Heterogeneous multi-node clusters with accelerators, such as GPUs, are increasingly the standard for HPC, where the MPI+OpenMP approach is commonly used for application development. However, this approach poses significant challenges for developers, especially in managing communication, synchronization, and load balancing across distributed nodes and accelerators. To address these challenges, this paper proposes the MPI Proxy Plugin (MPP), an extension of the LLVM OpenMP Offloading runtime that transparently offloads OpenMP target regions to remote accelerators via MPI. By abstracting communication and using the asynchronous mechanisms of MPI with C++20 coroutines, MPP provides a scalable alternative to MPI+OpenMP, enabling simpler development of heterogeneous HPC applications through the familiar OpenMP programming model. Experimental results show that MPP achieves excellent scalability. For compute-intensive proxy applications, it scales nearly linearly, reaching \(63\times \) speedup from 1 to 64 GPUs. While naive data transfers can degrade performance, this research reveals that extending OpenMP Target with collective operations (e.g., multi-device broadcast) simplifies development and improves performance, achieving up to \(10\times \) speedup in communication-bound benchmarks.