Workflow Batch Job Scheduling with Considering Task Dependencies
摘要
In recent years, High Performance Computing (HPC) systems have been used for scientific data analyses and simulations defined as workflows, leading to an increasing demand for efficient workflow execution on HPC systems. However, conventional HPC systems are optimized to execute a large number of independent jobs efficiently and face challenges such as excessive resource allocation and increased turnaround time when executing workflows consisting of multiple dependent tasks. To address this issue, this paper proposes a job scheduling method to reduce the turnaround time of workflows. Specifically, it separates the control of the job execution order from the control of the task execution order. Additionally, when controlling the task execution order, it assigns execution priorities while considering task dependencies. As a result, compared to conventional methods that do not control task execution order, the proposed method significantly reduces the turnaround time.