Accurate job runtime prediction is essential for efficient job scheduling, which in turn significantly contributes to system performance optimization in High Performance Computing (HPC) environments. However, the dynamic nature of HPC workloads, including fluctuations in job size, priority, and system conditions, poses considerable challenges for single prediction models. To address these challenges, this study proposes a novel Hierarchical Dynamic Ensemble Model (HDEM), featuring an ensemble-of-ensembles architecture that adapts to workload variations. HDEM integrates a prediction degradation detection mechanism to dynamically adjust the contributions of base learners. Evaluated on historical workload data from production HPC systems, the proposed method demonstrates superior performance compared to conventional models and provides more reliable runtime estimates. This approach not only enhances prediction accuracy but also ensures adaptability to evolving workload patterns in HPC environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

HDEM: A Hierarchical Dynamic Ensemble Model for Accurate Runtime Prediction on High Performance Computing Systems

  • Thanh Hoang Le Hai,
  • Huy Nguyen Tuan,
  • Bao Tran Dang,
  • Bao Vo Thuong,
  • Khoi Phan Tran Dinh,
  • Nam Thoai

摘要

Accurate job runtime prediction is essential for efficient job scheduling, which in turn significantly contributes to system performance optimization in High Performance Computing (HPC) environments. However, the dynamic nature of HPC workloads, including fluctuations in job size, priority, and system conditions, poses considerable challenges for single prediction models. To address these challenges, this study proposes a novel Hierarchical Dynamic Ensemble Model (HDEM), featuring an ensemble-of-ensembles architecture that adapts to workload variations. HDEM integrates a prediction degradation detection mechanism to dynamically adjust the contributions of base learners. Evaluated on historical workload data from production HPC systems, the proposed method demonstrates superior performance compared to conventional models and provides more reliable runtime estimates. This approach not only enhances prediction accuracy but also ensures adaptability to evolving workload patterns in HPC environments.