Optimizing the configuration parameters of Spark applications presents a challenging problem [10, 15] in cloud computing, with the optimization of Spark multi-query applications being the most challenging aspect. Multi-query applications consist of multiple queries, such as TPC-DS [27] and TPC-H [4]. As the amount of data increases, the optimization cost of multi-query applications will increase substantially, making these applications increasingly challenging to optimize. In this study, we propose an Ordinal and Parallel Optimization Method (OPOM). Specifically, we execute queries in parallel and design the Partition Execution Order of queries (ordinal), which can significantly reduce the optimization time cost of multi-query applications. In the benchmarks (TPC-DS and TPC-H) of three data sizes and the Huawei Cloud, OPOM achieved the highest Cost Reduction of optimization time compared to Cherrypick [2], ROBOTune [18], and LOCAT [37], with maximum reductions of 290.84%, 277.26%, and 231.99%, and average reductions of 239.25%, 228.54%, and 194.89%, respectively. The maximum speedups for OPOM, Cherrypick, ROBOTune, and LOCAT are \(12.13 \times \) , \(3.39 \times \) , \( 3.68 \times \) , \(3.73 \times \) , and the corresponding average speedups are \(8.97 \times \) , \(2.89 \times \) , \( 3.00 \times \) , \(3.05 \times \) , respectively.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

OPOM: The Ordinal and Parallel Optimization Method of Spark Multi-query Applications

  • Bingyu Guan

摘要

Optimizing the configuration parameters of Spark applications presents a challenging problem [10, 15] in cloud computing, with the optimization of Spark multi-query applications being the most challenging aspect. Multi-query applications consist of multiple queries, such as TPC-DS [27] and TPC-H [4]. As the amount of data increases, the optimization cost of multi-query applications will increase substantially, making these applications increasingly challenging to optimize. In this study, we propose an Ordinal and Parallel Optimization Method (OPOM). Specifically, we execute queries in parallel and design the Partition Execution Order of queries (ordinal), which can significantly reduce the optimization time cost of multi-query applications. In the benchmarks (TPC-DS and TPC-H) of three data sizes and the Huawei Cloud, OPOM achieved the highest Cost Reduction of optimization time compared to Cherrypick [2], ROBOTune [18], and LOCAT [37], with maximum reductions of 290.84%, 277.26%, and 231.99%, and average reductions of 239.25%, 228.54%, and 194.89%, respectively. The maximum speedups for OPOM, Cherrypick, ROBOTune, and LOCAT are \(12.13 \times \) , \(3.39 \times \) , \( 3.68 \times \) , \(3.73 \times \) , and the corresponding average speedups are \(8.97 \times \) , \(2.89 \times \) , \( 3.00 \times \) , \(3.05 \times \) , respectively.