Scholarlink

<p>Apache Spark has emerged as a leading in-memory big data processing framework, with cloud deployments offering scalability and cost efficiency. However, effective job scheduling in cloud-based Spark environments remains challenging due to resource heterogeneity, dynamic workloads, and strict deadline requirements. Existing schedulers often optimize isolated objectives, failing to address the complex trade-offs inherent in real-world deployments. To bridge this gap, we propose a distributional deep reinforcement learning (DRL) framework that jointly optimizes five key objectives: minimizing virtual machine (VM) cost, enhancing energy efficiency, ensuring deadline adherence, maximizing job throughput, and optimizing resource utilization. We implemented two DRL agents, Rainbow DQN and C51, within a Python-based simulation environment using TensorFlow, explicitly modeling Spark’s distributed execution patterns. Our experimental results demonstrate that Rainbow DQN achieves superior convergence stability and scheduling efficiency, reducing early-stage VM costs by 66%, improving CPU utilization by 12.5%, and enhancing deadline compliance by 4.2% compared to C51. While C51 exhibits faster initial convergence, Rainbow DQN delivers more robust long-term performance. These findings highlight the trade-offs between learning speed and optimization quality in DRL-based schedulers, providing actionable insights for cloud-based Spark deployments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Optimizing Spark job scheduling with distributional deep learning in cloud environments

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年10月27日

查看全文
  • Vishnu Prasad Verma,
  • Sumit Kumar,
  • Santosh Kumar,
  • Nenavath Srinivas Naik,
  • Rishav Dubey

摘要

Apache Spark has emerged as a leading in-memory big data processing framework, with cloud deployments offering scalability and cost efficiency. However, effective job scheduling in cloud-based Spark environments remains challenging due to resource heterogeneity, dynamic workloads, and strict deadline requirements. Existing schedulers often optimize isolated objectives, failing to address the complex trade-offs inherent in real-world deployments. To bridge this gap, we propose a distributional deep reinforcement learning (DRL) framework that jointly optimizes five key objectives: minimizing virtual machine (VM) cost, enhancing energy efficiency, ensuring deadline adherence, maximizing job throughput, and optimizing resource utilization. We implemented two DRL agents, Rainbow DQN and C51, within a Python-based simulation environment using TensorFlow, explicitly modeling Spark’s distributed execution patterns. Our experimental results demonstrate that Rainbow DQN achieves superior convergence stability and scheduling efficiency, reducing early-stage VM costs by 66%, improving CPU utilization by 12.5%, and enhancing deadline compliance by 4.2% compared to C51. While C51 exhibits faster initial convergence, Rainbow DQN delivers more robust long-term performance. These findings highlight the trade-offs between learning speed and optimization quality in DRL-based schedulers, providing actionable insights for cloud-based Spark deployments.