The network infrastructures currently face an onslaught of highly sophisticated and stealthy cyber threats in the increasingly more complex and connected digital world. Traditional rule-based detection systems rarely succeed in the identification of novel or evolving attack vectors, thus building the need for intelligent and adaptive anomaly detection frameworks. In this paper, we present a robust and scalable solution for network-based anomaly detection using the Explainable-Generalized Isolation Forest (EGIF) algorithm, which is well recognized for the identification of outliers in high-dimensional and imbalanced datasets representing network traffic. The GIF is an enhancement to the classical Isolation Forest methods that introduce more advanced candidate scoring mechanisms to better accommodate subtle anomalies and non-homogeneous data distributions. The pressing need in cybersecurity for explainability is solved by incorporating EGIF, a model-agnostic interpretability technique capable of providing justification. EGIF allows security analysts to act based on insights into feature-wise contributions to the anomaly scores, hence facilitating rapid Root Cause Analysis (RCA), making incident-response workflows more efficient, and satisfying explanations as mandated by regulation. In addition, this paper presents a cloud-native, real-time architecture hosted on Google Cloud Platform (GCP), deployed through Google Kubernetes Engine (GKE) yet cloud-agnostic at the application level. The architecture employs open-source technologies such as Apache Kafka, Apache Spark, Apache Airflow, and Prometheus/Grafana to ensure scalable data ingestion, processing, orchestration, and monitoring. This combination guarantees not only high performance and resilience but also real-time detection of incidents with interpretable reports. The system thus proposed represents a major enhancement to the security posture and trustworthiness of any automated network monitoring solution in highly dynamic environments, characteristic of modern enterprises.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Network Security: Anomaly Detection Using Generalized Isolation Forest and Explainable AI

  • Karan Alang,
  • Anirudh Khanna,
  • Suryaprakash Nalluri

摘要

The network infrastructures currently face an onslaught of highly sophisticated and stealthy cyber threats in the increasingly more complex and connected digital world. Traditional rule-based detection systems rarely succeed in the identification of novel or evolving attack vectors, thus building the need for intelligent and adaptive anomaly detection frameworks. In this paper, we present a robust and scalable solution for network-based anomaly detection using the Explainable-Generalized Isolation Forest (EGIF) algorithm, which is well recognized for the identification of outliers in high-dimensional and imbalanced datasets representing network traffic. The GIF is an enhancement to the classical Isolation Forest methods that introduce more advanced candidate scoring mechanisms to better accommodate subtle anomalies and non-homogeneous data distributions. The pressing need in cybersecurity for explainability is solved by incorporating EGIF, a model-agnostic interpretability technique capable of providing justification. EGIF allows security analysts to act based on insights into feature-wise contributions to the anomaly scores, hence facilitating rapid Root Cause Analysis (RCA), making incident-response workflows more efficient, and satisfying explanations as mandated by regulation. In addition, this paper presents a cloud-native, real-time architecture hosted on Google Cloud Platform (GCP), deployed through Google Kubernetes Engine (GKE) yet cloud-agnostic at the application level. The architecture employs open-source technologies such as Apache Kafka, Apache Spark, Apache Airflow, and Prometheus/Grafana to ensure scalable data ingestion, processing, orchestration, and monitoring. This combination guarantees not only high performance and resilience but also real-time detection of incidents with interpretable reports. The system thus proposed represents a major enhancement to the security posture and trustworthiness of any automated network monitoring solution in highly dynamic environments, characteristic of modern enterprises.