Scholarlink

<p>This study evaluates the performance of multi-node Hadoop clusters on Google Cloud Platform (GCP) compared to local configurations, using the HiBench benchmark suite. While prior research has explored Hadoop’s performance on individual setups, direct comparisons between cloud-based and on-premise multi-node clusters remain limited. This study fills that gap by analyzing the efficiency, scalability, and resource management of GCP clusters versus traditional local systems. The objective is to assess GCP’s capability in handling data-intensive workloads. The Terasort benchmark from HiBench was used to simulate real-world big data tasks. The experiments were conducted on clusters configured with a master node and two additional nodes, each utilizing the n2-standard-2 machine type with 2 vCPUs, 1 core, 8 GB of memory, and 100 GB of disk space. Moreover, the benchmarks were executed on datasets carefully chosen to represent a variety of data sizes–Tiny (39.30 KB), Small (3.28 MB), Large (328.50 MB), and Huge (3.28 GB). Key performance metrics–CPU utilization, disk I/O throughput, and memory usage–were compared across the two environments. Results show that GCP outperforms the local setup under the identical configurations tested, with 15% better CPU utilization, 20% higher disk I/O throughput, and 10% more efficient memory usage. GCP also demonstrated superior scalability compared to other cloud platforms, making it ideal for large-scale data processing. This research highlights the advantages of cloud-based solutions for big data. Future work could explore hybrid cloud architectures and advanced resource optimization for improved performance and cost efficiency, further enhancing the understanding of cloud versus local cluster performance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Performance Comparison of Cloud and Local Multi-node Hadoop Clusters Using HiBench Benchmarks

  • 期刊论文

  • 发表日期：

    2025年8月2日

查看全文
  • Harshit Kumar Chaubey,
  • Siri Arelli,
  • K. Mallikharjuna Rao

摘要

This study evaluates the performance of multi-node Hadoop clusters on Google Cloud Platform (GCP) compared to local configurations, using the HiBench benchmark suite. While prior research has explored Hadoop’s performance on individual setups, direct comparisons between cloud-based and on-premise multi-node clusters remain limited. This study fills that gap by analyzing the efficiency, scalability, and resource management of GCP clusters versus traditional local systems. The objective is to assess GCP’s capability in handling data-intensive workloads. The Terasort benchmark from HiBench was used to simulate real-world big data tasks. The experiments were conducted on clusters configured with a master node and two additional nodes, each utilizing the n2-standard-2 machine type with 2 vCPUs, 1 core, 8 GB of memory, and 100 GB of disk space. Moreover, the benchmarks were executed on datasets carefully chosen to represent a variety of data sizes–Tiny (39.30 KB), Small (3.28 MB), Large (328.50 MB), and Huge (3.28 GB). Key performance metrics–CPU utilization, disk I/O throughput, and memory usage–were compared across the two environments. Results show that GCP outperforms the local setup under the identical configurations tested, with 15% better CPU utilization, 20% higher disk I/O throughput, and 10% more efficient memory usage. GCP also demonstrated superior scalability compared to other cloud platforms, making it ideal for large-scale data processing. This research highlights the advantages of cloud-based solutions for big data. Future work could explore hybrid cloud architectures and advanced resource optimization for improved performance and cost efficiency, further enhancing the understanding of cloud versus local cluster performance.