EPIC-Q: Equivalent-Policy Invariant Comparison Enhanced Transfer Q-learning for Run-Time SoC Performance-Power Optimization
摘要
As power density becomes the main constraint of multicore systems, managing power consumption using DVFS while providing the desired performance becomes increasingly critical. Reinforcement learning (RL) performs significantly better than conventional methods in performance-power optimization under different hardware configurations and varying software applications. RL agents learn through trial-and-error by receiving rewards which is defined by an objective function (e.g. instructions-per-second (IPS)) within specified constraints (e.g. power budget). System and application requirements lead to changing objectives and constraints which in turn result in different reward functions. The RL agents adapt to these changing objectives and constraints (and hence reward functions). Equivalent-policy invariant comparison (EPIC) is a popular technique to evaluate different reward functions. EPIC provides a numerical score which quantifies the difference in two reward functions. In this work, we use this EPIC distance (score) to transfer knowledge and improve learning for changing reward functions. Experimental results using a DVFS enabled RISCV based system-on-chip implemented on an FPGA shows 16.2% lower power budget overshoots compared to a tabular Q-learning agent with direct transfer.