In cluster computing systems, memory limitations have become a critical bottleneck for big data classification. We propose a Stratified Asymptotic Sampling (SAS) method based on Hadoop Distributed File System (HDFS) storage mechanism to address this issue. SAS treats HDFS data blocks as sampling strata, implementing intra-block random sampling and block-level progressive learning strategy. A voxel-based d-dimensional Kolmogorov-Smirnov (vdKS) test serves as a stopping criterion to ensure sampling consistency. The method reduces memory demands by progressive processing, enabling fast sampling. The designed secondary indexing system stores only block numbers and line numbers of the sampled data, creating an efficient data retrieval mechanism. Experiments demonstrate that SAS significantly improves sampling efficiency when data size far exceeds cluster memory capacity, showing excellent performance by using a progressive enhancement classification framework with a Memory-Adaptive Progressive Ensemble (MAPE) strategy.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Resolving Memory Challenges in Cluster Computing Systems Via Stratified Asymptotic Sampling for Big Data Classification

  • Chenghao Wei,
  • Quan Li,
  • Chen Li,
  • PuKai Wang

摘要

In cluster computing systems, memory limitations have become a critical bottleneck for big data classification. We propose a Stratified Asymptotic Sampling (SAS) method based on Hadoop Distributed File System (HDFS) storage mechanism to address this issue. SAS treats HDFS data blocks as sampling strata, implementing intra-block random sampling and block-level progressive learning strategy. A voxel-based d-dimensional Kolmogorov-Smirnov (vdKS) test serves as a stopping criterion to ensure sampling consistency. The method reduces memory demands by progressive processing, enabling fast sampling. The designed secondary indexing system stores only block numbers and line numbers of the sampled data, creating an efficient data retrieval mechanism. Experiments demonstrate that SAS significantly improves sampling efficiency when data size far exceeds cluster memory capacity, showing excellent performance by using a progressive enhancement classification framework with a Memory-Adaptive Progressive Ensemble (MAPE) strategy.