Scholarlink

<p>Frequent itemset mining plays a critical role in knowledge discovery, particularly in the era of big data, and necessitates high-performance computing (HPC) solutions for feasible execution times and memory management. Traditional single-node algorithms, such as FP-Growth, encounter significant challenges when processing large-scale datasets due to excessive memory consumption and computational inefficiencies. To address these limitations, this study introduces two distributed adaptations of FP-Growth: Distributed FP-Growth (DFP-Growth) and Distributed Improved FP-Growth (DIFP-Growth) that leverage the power of parallel and distributed supercomputing frameworks. These methods leverage the Resilient Distributed Dataset (RDD) programming model to parallel processing capabilities, enhance scalability and performance in large-scale data processing across an Apache Spark cluster. DIFP-Growth introduces key optimizations, including vertical item grouping, single-insertion strategy, and <i>max</i>_<i>children</i> parameter for constructing the FP-Tree, reducing computational, communication and memory overhead to enhance parallel efficiency in a distributed memory environment. Experimental evaluations on large datasets, (Poker Hand, Susy, Higgs) demonstrate that the proposed methods outperform existing approaches such as PFP, DECLAT, and DATID in terms of execution time and memory efficiency. The results confirm that DIFP-Growth achieves superior performance by efficiently managing computational resources and minimizing redundant operations. For example, with min_supp = 10% on the SUSY dataset, DIFP-Growth method achieved 17.3% reduction in runtime and 89.7% reduction in memory usage compared to DATID, while also demonstrating a 1.18% runtime and 15.6% memory improvement over our own distributed baseline, DFP-Growth. These results underscore the critical requirement for HPC-driven approaches and highlight the effectiveness of our algorithmic innovations in massive datasets mining and provide a foundation for future research in scalable association rule mining.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Distributed improved FP-Growth with level-wise and memory-aware pruning for scalable frequent itemset mining on Apache Spark™

  • 期刊论文

  • 发表日期：

    2025年12月27日

查看全文
  • Masoud Barkhan,
  • Navid Khaledian,
  • Farough Ashkouti

摘要

Frequent itemset mining plays a critical role in knowledge discovery, particularly in the era of big data, and necessitates high-performance computing (HPC) solutions for feasible execution times and memory management. Traditional single-node algorithms, such as FP-Growth, encounter significant challenges when processing large-scale datasets due to excessive memory consumption and computational inefficiencies. To address these limitations, this study introduces two distributed adaptations of FP-Growth: Distributed FP-Growth (DFP-Growth) and Distributed Improved FP-Growth (DIFP-Growth) that leverage the power of parallel and distributed supercomputing frameworks. These methods leverage the Resilient Distributed Dataset (RDD) programming model to parallel processing capabilities, enhance scalability and performance in large-scale data processing across an Apache Spark cluster. DIFP-Growth introduces key optimizations, including vertical item grouping, single-insertion strategy, and max_children parameter for constructing the FP-Tree, reducing computational, communication and memory overhead to enhance parallel efficiency in a distributed memory environment. Experimental evaluations on large datasets, (Poker Hand, Susy, Higgs) demonstrate that the proposed methods outperform existing approaches such as PFP, DECLAT, and DATID in terms of execution time and memory efficiency. The results confirm that DIFP-Growth achieves superior performance by efficiently managing computational resources and minimizing redundant operations. For example, with min_supp = 10% on the SUSY dataset, DIFP-Growth method achieved 17.3% reduction in runtime and 89.7% reduction in memory usage compared to DATID, while also demonstrating a 1.18% runtime and 15.6% memory improvement over our own distributed baseline, DFP-Growth. These results underscore the critical requirement for HPC-driven approaches and highlight the effectiveness of our algorithmic innovations in massive datasets mining and provide a foundation for future research in scalable association rule mining.