Scholarlink

<p>The rapid development of cloud technologies has driven substantial growth in the scale and computational capacity of data centers. However, frequent workload failures under high-concurrency conditions have become a major performance bottleneck, causing resource waste and reduced operational efficiency. Meanwhile, evolving data center architectures, resource management policies, and workload characteristics have rendered existing failure analysis and prediction methods less effective and less interpretable, while traditional early-termination-based mitigation strategies often overlook the potential debugging value of failed workloads, negatively impacting user experience. To address these challenges, this study systematically analyzes workload failure patterns using the Google Cluster Trace (e-cluster) data across multiple dimensions, including job attributes, resource allocation and usage, job dependencies, and server characteristics. Interpretable failure rules are constructed using Rule-based Representation Learner (RRL), revealing that scheduling class and priority are among the key factors influencing failures, and that server instance failures exhibit a bimodal distribution. Based on these empirical findings, 51 features are extracted from the e-cluster data, and 14 key features are selected via random forest feature importance and Spearman correlation analysis to develop an early failure prediction framework based on XGBoost using the selected workload features. Experimental evaluation demonstrates that the framework achieves a weighted AUC of 0.9993 and a weighted F1 score of 0.9936 across different stages of the instance lifecycle. Furthermore, the prediction framework is integrated with an active cloning mechanism to mitigate the impact of predicted instance failures. Results show that the integrated mitigation mechanism achieves a median speedup of 26% for failed instances under low additional overhead.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Predicting and mitigating early job failures in cloud data centers

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月28日

查看全文
  • Ling He,
  • Zhaofang Luan,
  • Congfeng Jiang,
  • Christophe Cerin

摘要

The rapid development of cloud technologies has driven substantial growth in the scale and computational capacity of data centers. However, frequent workload failures under high-concurrency conditions have become a major performance bottleneck, causing resource waste and reduced operational efficiency. Meanwhile, evolving data center architectures, resource management policies, and workload characteristics have rendered existing failure analysis and prediction methods less effective and less interpretable, while traditional early-termination-based mitigation strategies often overlook the potential debugging value of failed workloads, negatively impacting user experience. To address these challenges, this study systematically analyzes workload failure patterns using the Google Cluster Trace (e-cluster) data across multiple dimensions, including job attributes, resource allocation and usage, job dependencies, and server characteristics. Interpretable failure rules are constructed using Rule-based Representation Learner (RRL), revealing that scheduling class and priority are among the key factors influencing failures, and that server instance failures exhibit a bimodal distribution. Based on these empirical findings, 51 features are extracted from the e-cluster data, and 14 key features are selected via random forest feature importance and Spearman correlation analysis to develop an early failure prediction framework based on XGBoost using the selected workload features. Experimental evaluation demonstrates that the framework achieves a weighted AUC of 0.9993 and a weighted F1 score of 0.9936 across different stages of the instance lifecycle. Furthermore, the prediction framework is integrated with an active cloning mechanism to mitigate the impact of predicted instance failures. Results show that the integrated mitigation mechanism achieves a median speedup of 26% for failed instances under low additional overhead.