Scholarlink

<p>As a critical information source in distributed systems and large-scale IoT infrastructures, log data serves as the foundation for fault diagnosis and security auditing. However, the explosive growth of unstructured and dynamically changing log data poses significant challenges for real-time big data analytics and system scalability. To overcome these performance bottlenecks, this paper proposes and optimizes an end-to-end log anomaly detection method tailored for large-scale log data processing. Specifically, to address template inconsistency across distributed environments, we propose a distributed Drain parsing algorithm featuring dynamic parse tree expansion and global incremental synchronization. Furthermore, we optimize the HDBSCAN algorithm by integrating the RTree spatial index and Boruvka minimum spanning tree (MST). This algorithmic breakthrough effectively eliminates computational bottlenecks in real-time processing and significantly enhances the capacity to capture non-spherical clusters. By proposing a fusion of cross-partition clustering and the GLOSH scoring mechanism, our method strongly supports intelligent analytics in distributed infrastructures. We validate the model on the Spark framework using HDFS, BGL, and OpenStack datasets. Experimental results show that our proposed method has good performance in terms of scalability, resolution efficiency and detection performance, making it highly relevant to modern Internet of Things and large-scale systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Distributed log anomaly detection method based on improved HDBSCAN

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月26日

查看全文
  • Xunyun Wu,
  • Jiabin Wang,
  • Xilong Lin,
  • Jiexuan Zhuang,
  • Hao Cheng

摘要

As a critical information source in distributed systems and large-scale IoT infrastructures, log data serves as the foundation for fault diagnosis and security auditing. However, the explosive growth of unstructured and dynamically changing log data poses significant challenges for real-time big data analytics and system scalability. To overcome these performance bottlenecks, this paper proposes and optimizes an end-to-end log anomaly detection method tailored for large-scale log data processing. Specifically, to address template inconsistency across distributed environments, we propose a distributed Drain parsing algorithm featuring dynamic parse tree expansion and global incremental synchronization. Furthermore, we optimize the HDBSCAN algorithm by integrating the RTree spatial index and Boruvka minimum spanning tree (MST). This algorithmic breakthrough effectively eliminates computational bottlenecks in real-time processing and significantly enhances the capacity to capture non-spherical clusters. By proposing a fusion of cross-partition clustering and the GLOSH scoring mechanism, our method strongly supports intelligent analytics in distributed infrastructures. We validate the model on the Spark framework using HDFS, BGL, and OpenStack datasets. Experimental results show that our proposed method has good performance in terms of scalability, resolution efficiency and detection performance, making it highly relevant to modern Internet of Things and large-scale systems.