Scholarlink

We present Cluster in the Cloud, a free and open-source tool to create scalable, heterogeneous batch clusters on public and private cloud resources. Research workflows often require varied hardware throughout their pipeline: this may be needing a lot of RAM for one part, needing GPUs for another, or building on a particular CPU architecture; heterogeneous hardware is more and more required for research. Being able to scale heterogeneously—elastically as the work demands—is a requirement for reducing time-to-science in a cost-effective way. Clusters can be created in minutes, allowing environments to be created in response to novel research ideas or collaborations. Cluster in the Cloud provides a personal, at-scale platform for interactive and workflow based computing, AI and ML with CI/CD pipelines which are becoming more integrated into research. It has been used for published research into virology [16], carbon sequestration [13], computer science [3] and therapeutics [2] and has been deployed around the world by many universities (including Bath, Bristol, Caltech, Dublin City, RHUL and Sheffield) on public cloud and integrated into on-premise confidential computing systems.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Cluster in the Cloud—Scalable, Heterogeneous Compute Clusters for HPC, HTC and AI in the Public Cloud or On-Premise

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Matt Williams,
  • Chris Edsall,
  • Christopher Woods,
  • Sadaf Alam

摘要

We present Cluster in the Cloud, a free and open-source tool to create scalable, heterogeneous batch clusters on public and private cloud resources. Research workflows often require varied hardware throughout their pipeline: this may be needing a lot of RAM for one part, needing GPUs for another, or building on a particular CPU architecture; heterogeneous hardware is more and more required for research. Being able to scale heterogeneously—elastically as the work demands—is a requirement for reducing time-to-science in a cost-effective way. Clusters can be created in minutes, allowing environments to be created in response to novel research ideas or collaborations. Cluster in the Cloud provides a personal, at-scale platform for interactive and workflow based computing, AI and ML with CI/CD pipelines which are becoming more integrated into research. It has been used for published research into virology [16], carbon sequestration [13], computer science [3] and therapeutics [2] and has been deployed around the world by many universities (including Bath, Bristol, Caltech, Dublin City, RHUL and Sheffield) on public cloud and integrated into on-premise confidential computing systems.