Proactive hard disk drive (HDD) failure prediction is critical for large-scale data centers to mitigate data loss and operational risks. However, existing machine learning approaches struggle to handle distribution discrepancies and severe class imbalance across heterogeneous disk models, often necessitating separate models for each disk type. To overcome these limitations, this paper proposes a novel distance-aware multi-task learning framework that aims to achieve two key objectives: 1) to improve prediction accuracy without data augmentation by alleviating class imbalance through cross-model knowledge transfer, and 2) to reduce the number of required models in heterogeneous storage environments. Specifically, we first develop a disk model similarity metric based on their Self-Monitoring Analysis, and Reporting Technology (S.M.A.R.T.) attribute distributions and failure patterns. Using this metric, we apply a clustering algorithm to group similar disk models into distinct clusters. Crucially, within each cluster, we formalize the failure prediction tasks for different disk models as a multi-task learning problem. To effectively solve this multi-task problem and enhance prediction performance through multi-perspective feature interactions, we introduce a Multi-gate Mixture-of-Experts (MMoE) network. Experimental results show that our framework achieves an average 3.1% improvement in G-MEAN, while drastically reducing the required models, highlighting the effectiveness of our approach for large-scale heterogeneous storage systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-task Learning for HDD Failure Prediction in Heterogeneous Storage Systems

  • Jixing Zhu,
  • Jiabao Zhao,
  • Fan Liu

摘要

Proactive hard disk drive (HDD) failure prediction is critical for large-scale data centers to mitigate data loss and operational risks. However, existing machine learning approaches struggle to handle distribution discrepancies and severe class imbalance across heterogeneous disk models, often necessitating separate models for each disk type. To overcome these limitations, this paper proposes a novel distance-aware multi-task learning framework that aims to achieve two key objectives: 1) to improve prediction accuracy without data augmentation by alleviating class imbalance through cross-model knowledge transfer, and 2) to reduce the number of required models in heterogeneous storage environments. Specifically, we first develop a disk model similarity metric based on their Self-Monitoring Analysis, and Reporting Technology (S.M.A.R.T.) attribute distributions and failure patterns. Using this metric, we apply a clustering algorithm to group similar disk models into distinct clusters. Crucially, within each cluster, we formalize the failure prediction tasks for different disk models as a multi-task learning problem. To effectively solve this multi-task problem and enhance prediction performance through multi-perspective feature interactions, we introduce a Multi-gate Mixture-of-Experts (MMoE) network. Experimental results show that our framework achieves an average 3.1% improvement in G-MEAN, while drastically reducing the required models, highlighting the effectiveness of our approach for large-scale heterogeneous storage systems.