Towards Efficient Scheduling in Large Clusters Leveraging the Small-World Network Model
摘要
The rapid expansion of data center clusters, coupled with a surge in short task traffic, poses significant challenges to traditional tree-structured schedulers, particularly centralized and shared-state systems. These challenges include computational bottlenecks at a few master nodes and the burdensome management of extensive cluster state information. In response, this paper investigates the potential of leveraging the small-world network model to significantly enhance cluster throughput and reduce task scheduling latency, providing an empirical basis for the design of our distributed scheduling framework named Beehive. Experimental results demonstrate the efficacy of the small-world network model in facilitating linear scalability of cluster throughput with cluster size, with 99% of tasks scheduling latency within 50 ms. Compared to conventional tree-structured schedulers, the small-world network topology offers substantial improvements in throughput and latency, especially for high-frequency, short tasks in large clusters. This study provides an empirical foundation for the adoption of small-world networks in advanced scheduling systems.