<p>In distributed processing environments and High-Performance Computing (HPC) systems, Hadoop Distributed File System (HDFS) is a popular open-source solution for storing and managing a massive number of files. However, both in traditional Hadoop-based data processing and in HPC environments integrating big data workloads, HDFS faces several issues when it comes to handling a large amount of small files. To overcome this drawback, we propose a new strategy CHAC (for Correlation and Hierarchical Ascending Clustering) dealing with the small files problem. The major contribution of our strategy is that considering correlations between small files is the criterion used in the merging process into large files. For this purpose, several criteria such as file size, requests number and requesting clients are taken into account through analyzing the user access pattern. In the current version of our proposal, a formula inspired from information retrieval is used to quantify the weight of each small file. Moreover, a Cosine similarity-based method coupled with a hierarchical ascending clustering algorithm is used as a shrewd grouping tool of correlated small files. To demonstrate the effectiveness of CHAC, a series of experiments were performed over ten distinct datasets. Compared with other solutions, the obtained results highlight that our proposal offers interesting performances for reducing the number of obtained large files as well as the NameNode memory consumption and consumes approximately 55% less memory than HPF. CHAC also optimizes the use of the DataNodes storage space by increasing the average disk utilization of data blocks. In most cases, our strategy uses about 98% of the data blocks size. Moreover, it reduces the storage time taken to store large files. As our solution is based on file correlations, an in-depth analysis of the quality of the obtained large files is carried out highlighting the effectiveness of CHAC.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Correlation aware solution for merging small files based on similarity and clustering in HDFS

  • Hanène Chettaoui,
  • Farah Hkiri

摘要

In distributed processing environments and High-Performance Computing (HPC) systems, Hadoop Distributed File System (HDFS) is a popular open-source solution for storing and managing a massive number of files. However, both in traditional Hadoop-based data processing and in HPC environments integrating big data workloads, HDFS faces several issues when it comes to handling a large amount of small files. To overcome this drawback, we propose a new strategy CHAC (for Correlation and Hierarchical Ascending Clustering) dealing with the small files problem. The major contribution of our strategy is that considering correlations between small files is the criterion used in the merging process into large files. For this purpose, several criteria such as file size, requests number and requesting clients are taken into account through analyzing the user access pattern. In the current version of our proposal, a formula inspired from information retrieval is used to quantify the weight of each small file. Moreover, a Cosine similarity-based method coupled with a hierarchical ascending clustering algorithm is used as a shrewd grouping tool of correlated small files. To demonstrate the effectiveness of CHAC, a series of experiments were performed over ten distinct datasets. Compared with other solutions, the obtained results highlight that our proposal offers interesting performances for reducing the number of obtained large files as well as the NameNode memory consumption and consumes approximately 55% less memory than HPF. CHAC also optimizes the use of the DataNodes storage space by increasing the average disk utilization of data blocks. In most cases, our strategy uses about 98% of the data blocks size. Moreover, it reduces the storage time taken to store large files. As our solution is based on file correlations, an in-depth analysis of the quality of the obtained large files is carried out highlighting the effectiveness of CHAC.