Scholarlink

<p>With the advent of the AI era and the success of large models like ChatGPT, investments in AI training for advanced large models have been increasing. As AI models rely on fundamental computing infrastructure, high-performance computing (HPC) environments—such as supercomputers—have grown increasingly large and complex. However, system faults due to increasing complexity and the number of nodes can degrade system availability. Therefore, minimizing fault detection time is crucial. To address this problem, we developed an innovative model—health-aware failure classification using bidirectional encoder representations from transformers (HABERT)—which leverages advanced representation learning to more accurately capture fault events through system logs. These logs contain event information from various components such as the OS, CPU, GPU, and power supply. This model was developed from 23,699 unique log patterns extracted from Samsung Electronics' SSC-21 system logs accumulated over 18&#xa0;months since 2021. Trained with domain and task adaptation using a RoBERTa-based model, HABERT achieved an 89.6% F1-score in failure detection. We integrated HABERT as the core fault detection engine into our comprehensive health-aware monitoring system (HAMS), which incorporates additional real-time log analysis and monitoring functionalities. By deploying HAMS during a 3-month AI training job on the SSC-21, the system improved availability from 95.9 to 97.8%, marking a substantial enhancement.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

HAMS: an AI-driven framework for real-time failure detection in HPC system logs

  • 期刊论文

  • 发表日期：

    2025年9月20日

查看全文
  • Byungwoo Bang,
  • Jaehoon Kim,
  • Uiseok Song,
  • Junyeon Lee,
  • Jeonghyeon Ma

摘要

With the advent of the AI era and the success of large models like ChatGPT, investments in AI training for advanced large models have been increasing. As AI models rely on fundamental computing infrastructure, high-performance computing (HPC) environments—such as supercomputers—have grown increasingly large and complex. However, system faults due to increasing complexity and the number of nodes can degrade system availability. Therefore, minimizing fault detection time is crucial. To address this problem, we developed an innovative model—health-aware failure classification using bidirectional encoder representations from transformers (HABERT)—which leverages advanced representation learning to more accurately capture fault events through system logs. These logs contain event information from various components such as the OS, CPU, GPU, and power supply. This model was developed from 23,699 unique log patterns extracted from Samsung Electronics' SSC-21 system logs accumulated over 18 months since 2021. Trained with domain and task adaptation using a RoBERTa-based model, HABERT achieved an 89.6% F1-score in failure detection. We integrated HABERT as the core fault detection engine into our comprehensive health-aware monitoring system (HAMS), which incorporates additional real-time log analysis and monitoring functionalities. By deploying HAMS during a 3-month AI training job on the SSC-21, the system improved availability from 95.9 to 97.8%, marking a substantial enhancement.