The rise of Generative AI has renewed interest in Deep Learning across academia and industry, attracting smaller research groups and non-IT companies eager to leverage Machine Learning (ML). However, high infrastructure costs often make AI adoption impractical. Democratizing access to High-Performance Computing (HPC) is key to overcoming this barrier, enabling broader ML adoption while reducing e-waste by integrating existing resources. DARE-ML (Democratized Accessible Resource Environment for Machine Learning) offers a framework to optimize resource allocation, lower energy use, and improve ML accessibility. By profiling models interactively in a heterogeneous, limited-GPU environment, DARE-ML collects key metadata—like training time and memory needs—before scheduling jobs. At its core, DARE-ML incorporates an efficient interactive profiling mechanism powered by ESN (Echo State Networks), enabling streamlined and resource-aware execution of deep learning tasks. Experiments in real and simulated settings show DARE-ML improves ML job scheduling, reducing energy use up to 80-fold and cutting both average job completion time and waiting time per user by 15%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DARE-ML: Democratized Accessible Resource Environment for Machine Learning in the SUPERCOM Platform

  • Matteo Mendula,
  • Caterina Leonelli,
  • Marco Miozzo,
  • Paolo Dini

摘要

The rise of Generative AI has renewed interest in Deep Learning across academia and industry, attracting smaller research groups and non-IT companies eager to leverage Machine Learning (ML). However, high infrastructure costs often make AI adoption impractical. Democratizing access to High-Performance Computing (HPC) is key to overcoming this barrier, enabling broader ML adoption while reducing e-waste by integrating existing resources. DARE-ML (Democratized Accessible Resource Environment for Machine Learning) offers a framework to optimize resource allocation, lower energy use, and improve ML accessibility. By profiling models interactively in a heterogeneous, limited-GPU environment, DARE-ML collects key metadata—like training time and memory needs—before scheduling jobs. At its core, DARE-ML incorporates an efficient interactive profiling mechanism powered by ESN (Echo State Networks), enabling streamlined and resource-aware execution of deep learning tasks. Experiments in real and simulated settings show DARE-ML improves ML job scheduling, reducing energy use up to 80-fold and cutting both average job completion time and waiting time per user by 15%.