Hyperparameter optimisation is a critical challenge in tuning Machine Learning algorithms for optimal performance on each problem. Basic hyperparameter optimisation methods like grid or random search face scalability limitations with large datasets and complex hyperparameter search spaces. Advanced techniques such as evolutionary heuristics and Bayesian optimisation, namely SMAC [15] and SMBOX [36] show promise though they still require considerable computational time. This study presents ZeroTune, a novel Zero-shot hyperparameter optimisation method that employs a pre-trained Random Forest model to predict near-optimal settings based on easily computable meta-parameters, such as the count of training samples and an estimation of the number of informative features. ZeroTune’s novel approach to optimising core Decision Tree hyperparameters, without the need for runtime learning, marks a departure from iterative methods like SMAC [15] or irace [29], offering rapid predictions of near-optimal configurations. Our empirical tests across diverse datasets demonstrate ZeroTune’s effectiveness, outperforming default parameter settings, between 96% to 100% of the time with no observed performance deterioration on more diverse dataset benchmarks of cases. The introduction of ZeroTune promises a substantial reduction in computational time, highlighting its potential to make efficient, scalable hyperparameter optimisation accessible to a wider array of machine learning projects.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Beyond Iterative Tuning: Zero-Shot Hyperparameter Optimisation for Decision Trees

  • Tarek Salhi,
  • John Woodward

摘要

Hyperparameter optimisation is a critical challenge in tuning Machine Learning algorithms for optimal performance on each problem. Basic hyperparameter optimisation methods like grid or random search face scalability limitations with large datasets and complex hyperparameter search spaces. Advanced techniques such as evolutionary heuristics and Bayesian optimisation, namely SMAC [15] and SMBOX [36] show promise though they still require considerable computational time. This study presents ZeroTune, a novel Zero-shot hyperparameter optimisation method that employs a pre-trained Random Forest model to predict near-optimal settings based on easily computable meta-parameters, such as the count of training samples and an estimation of the number of informative features. ZeroTune’s novel approach to optimising core Decision Tree hyperparameters, without the need for runtime learning, marks a departure from iterative methods like SMAC [15] or irace [29], offering rapid predictions of near-optimal configurations. Our empirical tests across diverse datasets demonstrate ZeroTune’s effectiveness, outperforming default parameter settings, between 96% to 100% of the time with no observed performance deterioration on more diverse dataset benchmarks of cases. The introduction of ZeroTune promises a substantial reduction in computational time, highlighting its potential to make efficient, scalable hyperparameter optimisation accessible to a wider array of machine learning projects.