The landscape of High Performance Computing (HPC) is rapidly evolving, resulting in significant increases in computational power. This advancement enables individuals to address tasks with larger workloads, where resources are allocated statically at the beginning. However, with static allocation, not all workloads can fully utilize the available compute power throughout the duration of the task. Furthermore, the ever-growing usage of HPC leads to more diverse workloads in terms of priority, ranging from time-critical to those that can be delayed. This creates a need for a scalable cluster management and job scheduling system (SCMJSS) capable of dynamically allocating resources, with the potential for co-scheduling [3] to best manage such diverse workloads. We introduce FlexiAlloc, a fast prototype version of scheduler with dynamic resource management framework aiming to address the aforementioned challenges and facilitate the study of scheduling strategies enabled by the malleability of HPC jobs. We also demonstrate the programming model, which is particularly well suited to capitalize on resource dynamicity. Our evaluation of dynamic job resource allocation and co-scheduling with FlexiAlloc shows an increase in throughput and \(\approx \) 4x faster execution compared to static allocation. In addition, we showcase how it handles dynamic workloads by preempting lower priority job to prioritize urgent job taking \(\approx \) 517 ms and the elasticity of a job across two clusters.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Resource Management Framework for Elastic Computing

  • Arjun Parab,
  • Amir Raoofy,
  • Josef Weidendorfer

摘要

The landscape of High Performance Computing (HPC) is rapidly evolving, resulting in significant increases in computational power. This advancement enables individuals to address tasks with larger workloads, where resources are allocated statically at the beginning. However, with static allocation, not all workloads can fully utilize the available compute power throughout the duration of the task. Furthermore, the ever-growing usage of HPC leads to more diverse workloads in terms of priority, ranging from time-critical to those that can be delayed. This creates a need for a scalable cluster management and job scheduling system (SCMJSS) capable of dynamically allocating resources, with the potential for co-scheduling [3] to best manage such diverse workloads. We introduce FlexiAlloc, a fast prototype version of scheduler with dynamic resource management framework aiming to address the aforementioned challenges and facilitate the study of scheduling strategies enabled by the malleability of HPC jobs. We also demonstrate the programming model, which is particularly well suited to capitalize on resource dynamicity. Our evaluation of dynamic job resource allocation and co-scheduling with FlexiAlloc shows an increase in throughput and \(\approx \) 4x faster execution compared to static allocation. In addition, we showcase how it handles dynamic workloads by preempting lower priority job to prioritize urgent job taking \(\approx \) 517 ms and the elasticity of a job across two clusters.