Multi-task Learning for HDD Failure Prediction in Heterogeneous Storage Systems
摘要
Proactive hard disk drive (HDD) failure prediction is critical for large-scale data centers to mitigate data loss and operational risks. However, existing machine learning approaches struggle to handle distribution discrepancies and severe class imbalance across heterogeneous disk models, often necessitating separate models for each disk type. To overcome these limitations, this paper proposes a novel distance-aware multi-task learning framework that aims to achieve two key objectives: 1) to improve prediction accuracy without data augmentation by alleviating class imbalance through cross-model knowledge transfer, and 2) to reduce the number of required models in heterogeneous storage environments. Specifically, we first develop a disk model similarity metric based on their Self-Monitoring Analysis, and Reporting Technology (S.M.A.R.T.) attribute distributions and failure patterns. Using this metric, we apply a clustering algorithm to group similar disk models into distinct clusters. Crucially, within each cluster, we formalize the failure prediction tasks for different disk models as a multi-task learning problem. To effectively solve this multi-task problem and enhance prediction performance through multi-perspective feature interactions, we introduce a Multi-gate Mixture-of-Experts (MMoE) network. Experimental results show that our framework achieves an average 3.1% improvement in G-MEAN, while drastically reducing the required models, highlighting the effectiveness of our approach for large-scale heterogeneous storage systems.