High-performance computing (HPC) environments require precise resource management and job scheduling to optimize system performance. SLURM, a widely used job scheduler, faces limitations in its native ability to predict job runtime, often resulting in suboptimal resource allocation and prolonged wait times. This study introduces BOSER, a novel machine-learning algorithm developed by the authors, designed to significantly improve SLURM’s predictive accuracy for job runtime estimation. BOSER leverages advanced ensemble learning techniques, integrating diverse base learners such as XGBoost, LightGBM, CatBoost, AdaBoost, and Random Forest, within an innovative stacking architecture. Unlike traditional stacking ensembles that rely on simpler meta-learners (e.g., Linear Regression), BOSER employs a VotingRegressor framework using ElasticNet, Ridge, and Lasso, achieving a perfect \(R^2\) of 1.0 and near-perfect alignment with real-world values. The algorithm also incorporates Bayesian optimization for hyperparameter tuning and advanced feature-engineering techniques (e.g., interaction terms and logarithmic transformations) to ensure robust model performance and generalizability across various datasets. We have developed a new plugin to integrate BOSER with SLURM, enhancing the scheduler’s job scheduling and resource allocation efficiency. The improvements in job runtime prediction help optimize overall system throughput, offering a scalable, data-driven solution to address the challenges of predictive modeling in HPC resource management, maximizing computational efficiency and minimizing resource waste.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Machine Learning-Based Plugin for SLURM: Improving Job Scheduling Through Ensemble Learning

  • Mahdi Rezaei,
  • Alexey Salnikov

摘要

High-performance computing (HPC) environments require precise resource management and job scheduling to optimize system performance. SLURM, a widely used job scheduler, faces limitations in its native ability to predict job runtime, often resulting in suboptimal resource allocation and prolonged wait times. This study introduces BOSER, a novel machine-learning algorithm developed by the authors, designed to significantly improve SLURM’s predictive accuracy for job runtime estimation. BOSER leverages advanced ensemble learning techniques, integrating diverse base learners such as XGBoost, LightGBM, CatBoost, AdaBoost, and Random Forest, within an innovative stacking architecture. Unlike traditional stacking ensembles that rely on simpler meta-learners (e.g., Linear Regression), BOSER employs a VotingRegressor framework using ElasticNet, Ridge, and Lasso, achieving a perfect \(R^2\) of 1.0 and near-perfect alignment with real-world values. The algorithm also incorporates Bayesian optimization for hyperparameter tuning and advanced feature-engineering techniques (e.g., interaction terms and logarithmic transformations) to ensure robust model performance and generalizability across various datasets. We have developed a new plugin to integrate BOSER with SLURM, enhancing the scheduler’s job scheduling and resource allocation efficiency. The improvements in job runtime prediction help optimize overall system throughput, offering a scalable, data-driven solution to address the challenges of predictive modeling in HPC resource management, maximizing computational efficiency and minimizing resource waste.