A disaggregated Query Engine can make use of multiple nodes and accelerators available in a cloud or cluster environment. In a production environment, it is important to control the use of the computing resources assigned to a task to avoid unnecessarily exhausting the computational assets of a cluster and incurring cost overruns. Sometimes, tuning these resources against the workload must be done manually, which can result in an overestimation of the amount and types of compute nodes needed or an underestimation, depending on the workload that the system presents at a given time. We propose the creation of the Dynamic Resource Allocator (DRA) for Disaggregated Query Engines, an artificial intelligence agent in charge of reserving or freeing computational resources for the execution of queries. A key feature of this agent is the use of Reinforcement Learning (RL). An agent will be able to analyze the job execution time and node usage to learn a policy that leads to better execution time and resource allocation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Resource Allocation in Disaggregated Query Engines Using Deep Reinforcement Learning

  • Ivan D. Conde-Sarmiento,
  • Manuel Rodriguez-Martinez

摘要

A disaggregated Query Engine can make use of multiple nodes and accelerators available in a cloud or cluster environment. In a production environment, it is important to control the use of the computing resources assigned to a task to avoid unnecessarily exhausting the computational assets of a cluster and incurring cost overruns. Sometimes, tuning these resources against the workload must be done manually, which can result in an overestimation of the amount and types of compute nodes needed or an underestimation, depending on the workload that the system presents at a given time. We propose the creation of the Dynamic Resource Allocator (DRA) for Disaggregated Query Engines, an artificial intelligence agent in charge of reserving or freeing computational resources for the execution of queries. A key feature of this agent is the use of Reinforcement Learning (RL). An agent will be able to analyze the job execution time and node usage to learn a policy that leads to better execution time and resource allocation.