Going into the exascale era, there is an increasing demand to test and evaluate HPC tooling for such large scale computing systems. This goes beyond the aspects of scheduling performance, and also includes the development, testing and evaluation of workload manager plugins and profiling tools in the exascale. Current Slurm workload manager simulators, which are tailored for the former aspect, exhibit limitations in addressing the latter aspects. This paper outlines the detailed steps for setting up an exascale Slurm instance with virtual nodes, enabling the simulation of 10,000 nodes on a single 8-core machine. Notably, no modifications to the source code of Slurm itself are required, ensuring straightforward portability to new releases. The paper also introduces and evaluates a software tool named WOGE (Workload Generator and Evaluator). WOGE facilitates the automatic randomised generation and accelerated execution of workloads in the form of directed acyclic job graphs (job DAGs). To the best of the authors’ knowledge, this paper represents the first comprehensive documentation on establishing a Slurm testing and evaluation environment of this scale, and the incorporation of job DAGs in this context is unparalleled in the existing literature.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Exascale Slurm Testing and Evaluation Environment Utilising Generated DAG Workloads

  • Laslo Hunhold,
  • Stefan Wesner

摘要

Going into the exascale era, there is an increasing demand to test and evaluate HPC tooling for such large scale computing systems. This goes beyond the aspects of scheduling performance, and also includes the development, testing and evaluation of workload manager plugins and profiling tools in the exascale. Current Slurm workload manager simulators, which are tailored for the former aspect, exhibit limitations in addressing the latter aspects. This paper outlines the detailed steps for setting up an exascale Slurm instance with virtual nodes, enabling the simulation of 10,000 nodes on a single 8-core machine. Notably, no modifications to the source code of Slurm itself are required, ensuring straightforward portability to new releases. The paper also introduces and evaluates a software tool named WOGE (Workload Generator and Evaluator). WOGE facilitates the automatic randomised generation and accelerated execution of workloads in the form of directed acyclic job graphs (job DAGs). To the best of the authors’ knowledge, this paper represents the first comprehensive documentation on establishing a Slurm testing and evaluation environment of this scale, and the incorporation of job DAGs in this context is unparalleled in the existing literature.