Hierarchical Dependency-Aware Scheduling for Distributed Stream Computing Systems
摘要
Scheduling strategies play a crucial role in distributed stream computing systems. Many state-of-the-art works do not sufficiently consider differentiated communication costs between computing nodes, leading to increased communication latency, thereby degrading the overall system latency. To overcome these limitations, we propose Hd-Stream, which is a hierarchical dependency-aware scheduling mechanism for distributed stream systems. Our Hd-Stream mechanism comprises: (1) The dependency relationships of stream application tasks and computing nodes are hierarchically quantified by constructing a task dependency model and a resource dependency model. (2) A hierarchical dependency-aware scheduling algorithm with the comprehensive consideration of computational resource dependencies and task instance dependencies is proposed for reducing communication latency. (3) Multiple metrics are evaluated, including system latency, system throughput, and resource utilization in real distributed stream computing scenarios. Our experimental results demonstrate that Hd-Stream reduces system latency by 42% and achieves efficient resource utilization.