Scheduling strategies play a crucial role in distributed stream computing systems. Many state-of-the-art works do not sufficiently consider differentiated communication costs between computing nodes, leading to increased communication latency, thereby degrading the overall system latency. To overcome these limitations, we propose Hd-Stream, which is a hierarchical dependency-aware scheduling mechanism for distributed stream systems. Our Hd-Stream mechanism comprises: (1) The dependency relationships of stream application tasks and computing nodes are hierarchically quantified by constructing a task dependency model and a resource dependency model. (2) A hierarchical dependency-aware scheduling algorithm with the comprehensive consideration of computational resource dependencies and task instance dependencies is proposed for reducing communication latency. (3) Multiple metrics are evaluated, including system latency, system throughput, and resource utilization in real distributed stream computing scenarios. Our experimental results demonstrate that Hd-Stream reduces system latency by 42% and achieves efficient resource utilization.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hierarchical Dependency-Aware Scheduling for Distributed Stream Computing Systems

  • Yinuo Fan,
  • Dawei Sun,
  • Shuaiyi Zou,
  • Jonathan Kua,
  • Rajkumar Buyya

摘要

Scheduling strategies play a crucial role in distributed stream computing systems. Many state-of-the-art works do not sufficiently consider differentiated communication costs between computing nodes, leading to increased communication latency, thereby degrading the overall system latency. To overcome these limitations, we propose Hd-Stream, which is a hierarchical dependency-aware scheduling mechanism for distributed stream systems. Our Hd-Stream mechanism comprises: (1) The dependency relationships of stream application tasks and computing nodes are hierarchically quantified by constructing a task dependency model and a resource dependency model. (2) A hierarchical dependency-aware scheduling algorithm with the comprehensive consideration of computational resource dependencies and task instance dependencies is proposed for reducing communication latency. (3) Multiple metrics are evaluated, including system latency, system throughput, and resource utilization in real distributed stream computing scenarios. Our experimental results demonstrate that Hd-Stream reduces system latency by 42% and achieves efficient resource utilization.