In distributed computing frameworks, data skew is one of the main factors affecting performance improvement. The essence of this problem is that there are differences in the computing load of partition tasks, resulting in node load imbalance, especially in heterogeneous scenarios. This paper proposes a data skew greedy optimization strategy (HSDSGO) based on heterogeneous Spark to solve the data skew problem, which comprehensively considers differences in node performance and data locality. Firstly, we use an improved reservoir sampling algorithm to sample the input data in the Map phase, obtaining the key distribution in the entire dataset. Secondly, the capacity size of mapping partition tasks is determined based on the evaluation of node computational capabilities. Finally, the data is partitioned using a best-fit decreasing algorithm. Experimental results demonstrate that the HSDSGO algorithm not only reduces task execution time but also maximizes the utilization of cluster resources.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Data Skew Greedy Optimization Strategy in Spark Heterogeneous Clusters

  • Chen Huang,
  • Xiaoyong Tang

摘要

In distributed computing frameworks, data skew is one of the main factors affecting performance improvement. The essence of this problem is that there are differences in the computing load of partition tasks, resulting in node load imbalance, especially in heterogeneous scenarios. This paper proposes a data skew greedy optimization strategy (HSDSGO) based on heterogeneous Spark to solve the data skew problem, which comprehensively considers differences in node performance and data locality. Firstly, we use an improved reservoir sampling algorithm to sample the input data in the Map phase, obtaining the key distribution in the entire dataset. Secondly, the capacity size of mapping partition tasks is determined based on the evaluation of node computational capabilities. Finally, the data is partitioned using a best-fit decreasing algorithm. Experimental results demonstrate that the HSDSGO algorithm not only reduces task execution time but also maximizes the utilization of cluster resources.