Frequent itemset mining and association rule generation are common topics in mining massive datasets or big data, the purpose of which is to discover itemsets bought together with a significant proportion. Traditional algorithms are mostly implemented and evaluated as single-machine in-memory programs that are incapable of large-scale systems. In the study, the authors contrast the performance of two algorithms for frequent itemset mining, including A-Priori and Park-Chen-Yu (PCY), in the Hadoop Distribution File System (HDFS) using the PySpark framework. Empirical results in practical datasets emphasize the remarkable performance of PCY thanks to eliminating non-frequent pairs using buckets of candidate ones.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Comparison of Frequent Itemset Mining Algorithms in Distributed File Systems Using PySpark: A-Priori Versus PCY

  • Luat Tran,
  • Truc Cao,
  • Tien Nguyen,
  • Hung Nguyen,
  • Thanh-An Nguyen

摘要

Frequent itemset mining and association rule generation are common topics in mining massive datasets or big data, the purpose of which is to discover itemsets bought together with a significant proportion. Traditional algorithms are mostly implemented and evaluated as single-machine in-memory programs that are incapable of large-scale systems. In the study, the authors contrast the performance of two algorithms for frequent itemset mining, including A-Priori and Park-Chen-Yu (PCY), in the Hadoop Distribution File System (HDFS) using the PySpark framework. Empirical results in practical datasets emphasize the remarkable performance of PCY thanks to eliminating non-frequent pairs using buckets of candidate ones.