Log anomaly detection is crucial for the security of distributed computing systems. Although existing methods can capture the statistical characteristics and explicit timing patterns of log events, there are still two major challenges for semi-structured log files: 1) Complex topological dependencies between log events and implicit associations with multi-dimensional semantics of key fields have not been fully modeled, and anomalous patterns are easily overwhelmed by high-frequency normal events. 2) Dynamic log formats require field extraction relying on manual rules or supervised learning, which is difficult to adapt to operation and maintenance (O&M) requirements in low resource scenarios. To address these issues, we propose an unsupervised log anomaly detection framework based on graph neural networks (GNN). Specifically, key fields (e.g., anomalous IPs, failed APIs) are automatically extracted using prompt-based few-shot learning, then a weighted directed graph model fusing semantic embedding and temporal dependency is constructed to fully characterize the dynamic interaction patterns among system components. Moreover, global anomaly identification across events is achieved by co-optimizing graph representation learning and anomaly detection objectives based on one-class directed graph convolutional networks. Experimental results show that our method performs remarkably on multiple benchmark datasets and exhibits excellent generalization capabilities for unseen log templates, improving distributed system security.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Unsupervised Learning Log Anomaly Detection Method Based on Graph Neural Network

  • Xianlang Hu,
  • Guangsheng Feng,
  • Xinling Huang,
  • Xiangying Kong,
  • Hongwu Lv

摘要

Log anomaly detection is crucial for the security of distributed computing systems. Although existing methods can capture the statistical characteristics and explicit timing patterns of log events, there are still two major challenges for semi-structured log files: 1) Complex topological dependencies between log events and implicit associations with multi-dimensional semantics of key fields have not been fully modeled, and anomalous patterns are easily overwhelmed by high-frequency normal events. 2) Dynamic log formats require field extraction relying on manual rules or supervised learning, which is difficult to adapt to operation and maintenance (O&M) requirements in low resource scenarios. To address these issues, we propose an unsupervised log anomaly detection framework based on graph neural networks (GNN). Specifically, key fields (e.g., anomalous IPs, failed APIs) are automatically extracted using prompt-based few-shot learning, then a weighted directed graph model fusing semantic embedding and temporal dependency is constructed to fully characterize the dynamic interaction patterns among system components. Moreover, global anomaly identification across events is achieved by co-optimizing graph representation learning and anomaly detection objectives based on one-class directed graph convolutional networks. Experimental results show that our method performs remarkably on multiple benchmark datasets and exhibits excellent generalization capabilities for unseen log templates, improving distributed system security.