Apache Flink must scale its computational resources at run-time to comply with the real-time response requirements of fast-paced and changing workloads. TALOS is a task autoscaler designed to optimize Flink jobs’ performance while minimizing infrastructure usage costs in the cloud. Most autoscaling methods solve the resource adaptation problem by allocating new resources to the entire Flink job (i.e., pipeline). These solutions are suboptimal since not all tasks are equally stressed and do not need to be scaled, leading to over or under-provisioning of resources. TALOS monitors each task individually to decide how to scale the task based on its own data processing needs, without being affected by the performance of other upstream or downstream tasks. TALOS provides a better performance-to-cost ratio compared to the state-of-the-art Autoscaler of Flink Kubernetes Operator. Both agents are tested on sophisticated workloads running a click fraud detection application for several hours.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TALOS: Task Level Autoscaler for Apache Flink

  • Ourania Ntouni,
  • Euripides G. M. Petrakis

摘要

Apache Flink must scale its computational resources at run-time to comply with the real-time response requirements of fast-paced and changing workloads. TALOS is a task autoscaler designed to optimize Flink jobs’ performance while minimizing infrastructure usage costs in the cloud. Most autoscaling methods solve the resource adaptation problem by allocating new resources to the entire Flink job (i.e., pipeline). These solutions are suboptimal since not all tasks are equally stressed and do not need to be scaled, leading to over or under-provisioning of resources. TALOS monitors each task individually to decide how to scale the task based on its own data processing needs, without being affected by the performance of other upstream or downstream tasks. TALOS provides a better performance-to-cost ratio compared to the state-of-the-art Autoscaler of Flink Kubernetes Operator. Both agents are tested on sophisticated workloads running a click fraud detection application for several hours.