Scholarlink

<p>The increasing complexity of high-performance computing (HPC) systems, particularly those utilizing a multi-GPU cluster architecture, presents significant challenges for developers. Low-level, vendor-specific application programming interfaces (APIs) often require deep technical knowledge, complicating maintenance and reducing performance portability. This complexity is exacerbated when large data sizes or bandwidth requirements necessitate data compression. Integrating such end-to-end compression into existing codebases dealing with distributed memory multi-GPU clusters is often labor-intensive and error-prone, particularly when flexibility in the choice of compression algorithm and implementation is desired. In this work, we propose a high-level, user-friendly end-to-end data compression API integrated into the Celerity runtime system, built on the SYCL programming model. By abstracting the complexities of integrating various compression algorithms, we aim to enhance data transfer and storage efficiency, without a large cost in development complexity. Our API supports multiple compression types and memory layouts, enabling developers to leverage compression without extensive modifications to their existing codebases. We evaluate this API and its prototype implementation in various benchmarks, demonstrating its effectiveness in reducing storage and bandwidth requirements, while maintaining high performance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A High-Level API for End-to-End Data Compression in Multi-GPU Cluster Applications

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月16日

查看全文
  • Gabriel Mitterrutzner,
  • Peter Thoman,
  • Philipp Gschwandtner

摘要

The increasing complexity of high-performance computing (HPC) systems, particularly those utilizing a multi-GPU cluster architecture, presents significant challenges for developers. Low-level, vendor-specific application programming interfaces (APIs) often require deep technical knowledge, complicating maintenance and reducing performance portability. This complexity is exacerbated when large data sizes or bandwidth requirements necessitate data compression. Integrating such end-to-end compression into existing codebases dealing with distributed memory multi-GPU clusters is often labor-intensive and error-prone, particularly when flexibility in the choice of compression algorithm and implementation is desired. In this work, we propose a high-level, user-friendly end-to-end data compression API integrated into the Celerity runtime system, built on the SYCL programming model. By abstracting the complexities of integrating various compression algorithms, we aim to enhance data transfer and storage efficiency, without a large cost in development complexity. Our API supports multiple compression types and memory layouts, enabling developers to leverage compression without extensive modifications to their existing codebases. We evaluate this API and its prototype implementation in various benchmarks, demonstrating its effectiveness in reducing storage and bandwidth requirements, while maintaining high performance.