Scholarlink

<p>Log analysis is vital in Artificial Intelligence for IT Operations (AIOps) and plays a crucial role in ensuring software reliability and system stability. However, challenges such as the absence of comprehensive evaluation standards, inconsistencies in benchmarking practices, and limited exploration of Large Language Models (LLMs) in log-related tasks persist. To address these issues, we introduce <i>LogEval</i>, a comprehensive benchmark designed to systematically evaluate LLMs’ performance across four key log analysis tasks: log parsing, log anomaly detection, log fault diagnosis, and log summarization. <i>LogEval</i> systematically tackles these challenges through the following aspects: (i) it incorporates 4,000 publicly available log entries, spanning diverse tasks and providing a strong foundation for evaluating LLM performance; (ii) it utilizes standardized prompts in both English and Chinese to ensure consistent and objective evaluations, this benchmark covers two experimental paradigms: Naive question-answering (Q&amp;A) and self-consistency (SC) Q&amp;A, under both zero-shot and few-shot settings, while also considering inference efficiency and average token usage; (iii) it features an open-source, continuously updated platform (<a href="https://nkcs.iops.ai/LogEval/">https://nkcs.iops.ai/LogEval/</a>) that integrates new LLMs and user-uploaded production data, fostering reproducibility and adaptability in performance comparisons. The experimental results provide valuable insights into the varying strengths of LLMs across different tasks, highlighting opportunities for further optimization and innovation for LLMs in log analysis. Our code repository is available at <a href="https://github.com/LinDuoming/LogEval">https://github.com/LinDuoming/LogEval</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LogEval: A comprehensive benchmark suite for LLMs in log analysis

  • 期刊论文

  • 发表日期：

    2025年10月10日

查看全文
  • Tianyu Cui,
  • Shiyu Ma,
  • Ziang Chen,
  • Tong Xiao,
  • Chenyu Zhao,
  • Shimin Tao,
  • Yilun Liu,
  • Shenglin Zhang,
  • Duoming Lin,
  • Changchang Liu,
  • Yuzhe Cai,
  • Weibin Meng,
  • Yongqian Sun,
  • Dan Pei

摘要

Log analysis is vital in Artificial Intelligence for IT Operations (AIOps) and plays a crucial role in ensuring software reliability and system stability. However, challenges such as the absence of comprehensive evaluation standards, inconsistencies in benchmarking practices, and limited exploration of Large Language Models (LLMs) in log-related tasks persist. To address these issues, we introduce LogEval, a comprehensive benchmark designed to systematically evaluate LLMs’ performance across four key log analysis tasks: log parsing, log anomaly detection, log fault diagnosis, and log summarization. LogEval systematically tackles these challenges through the following aspects: (i) it incorporates 4,000 publicly available log entries, spanning diverse tasks and providing a strong foundation for evaluating LLM performance; (ii) it utilizes standardized prompts in both English and Chinese to ensure consistent and objective evaluations, this benchmark covers two experimental paradigms: Naive question-answering (Q&A) and self-consistency (SC) Q&A, under both zero-shot and few-shot settings, while also considering inference efficiency and average token usage; (iii) it features an open-source, continuously updated platform (https://nkcs.iops.ai/LogEval/) that integrates new LLMs and user-uploaded production data, fostering reproducibility and adaptability in performance comparisons. The experimental results provide valuable insights into the varying strengths of LLMs across different tasks, highlighting opportunities for further optimization and innovation for LLMs in log analysis. Our code repository is available at https://github.com/LinDuoming/LogEval.