DARE-ML: Democratized Accessible Resource Environment for Machine Learning in the SUPERCOM Platform
摘要
The rise of Generative AI has renewed interest in Deep Learning across academia and industry, attracting smaller research groups and non-IT companies eager to leverage Machine Learning (ML). However, high infrastructure costs often make AI adoption impractical. Democratizing access to High-Performance Computing (HPC) is key to overcoming this barrier, enabling broader ML adoption while reducing e-waste by integrating existing resources. DARE-ML (Democratized Accessible Resource Environment for Machine Learning) offers a framework to optimize resource allocation, lower energy use, and improve ML accessibility. By profiling models interactively in a heterogeneous, limited-GPU environment, DARE-ML collects key metadata—like training time and memory needs—before scheduling jobs. At its core, DARE-ML incorporates an efficient interactive profiling mechanism powered by ESN (Echo State Networks), enabling streamlined and resource-aware execution of deep learning tasks. Experiments in real and simulated settings show DARE-ML improves ML job scheduling, reducing energy use up to 80-fold and cutting both average job completion time and waiting time per user by 15%.