Modern workflows are increasingly data and event-driven, particularly with the growing integration of AI/ML tasks. Static workflows lack the flexibility to adapt at runtime in response to dynamic, event-driven tasks or fluctuating system demands, leading to suboptimal resource utilization and reduced efficiency. Elastic workflows, which can dynamically adjust resource usage based on system availability and workload demands, can enhance computational efficiency through intelligent resource management strategies. This paper presents the design and implementation of an elastic workflow framework for high-performance computing (HPC), highlighting key challenges and design considerations. Our framework is based on an elastic Parsl workflow manager and a custom elastic resource manager, both leveraging capabilities of an implementation of the Process Management Interface for Exscale (PMIx) API. Through real-world and synthetic workflow case studies, we demonstrate improved resource utilization and reduced execution times, showcasing the benefits of elasticity in HPC workflows.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enabling Elasticity in Scientific Workflows for High-Performance Computing Systems

  • Rajat Bhattarai,
  • Howard Pritchard,
  • Sheikh Ghafoor

摘要

Modern workflows are increasingly data and event-driven, particularly with the growing integration of AI/ML tasks. Static workflows lack the flexibility to adapt at runtime in response to dynamic, event-driven tasks or fluctuating system demands, leading to suboptimal resource utilization and reduced efficiency. Elastic workflows, which can dynamically adjust resource usage based on system availability and workload demands, can enhance computational efficiency through intelligent resource management strategies. This paper presents the design and implementation of an elastic workflow framework for high-performance computing (HPC), highlighting key challenges and design considerations. Our framework is based on an elastic Parsl workflow manager and a custom elastic resource manager, both leveraging capabilities of an implementation of the Process Management Interface for Exscale (PMIx) API. Through real-world and synthetic workflow case studies, we demonstrate improved resource utilization and reduced execution times, showcasing the benefits of elasticity in HPC workflows.