Scholarlink

<p>The rapid development of technology has led to a continuous increase in data dimensions, and the detection of high-dimensional data growing from tens of dimensions to millions of dimensions. As a result, the negative impact of the dimensional curse on anomaly detection has become increasingly remarkable. Although subspace research has been quite extensive, it has been difficult to extend to ultra-high-dimensional (UHD) datasets. To address this issue, this paper proposes the VAE-FastGA subspace search model, which is based on the idea of feature selection and aims to efficiently screen out anomalous subspaces from weakly correlated UHD data. The model integrates variational autoencoder (VAE) networks with accelerated genetic algorithms (FastGA), using VAE reconstruction to guide the search process of the genetic algorithm. An acceleration mechanism is designed to increase search efficiency by discarding invalid attributes during genetic operations and iteratively preserving anomalous subspaces. To mitigate the effects of sample sparsity, a probabilistic statistical model is introduced to optimize the anomalous subspaces. Finally, the idea of hyperdimensional spheres is integrated to improve the detection of marginal anomalies. The experimental sections organize precision experiments based on the KDD-CUP’99, MNIST, and UCSD datasets, perform subspace evaluations based on the MNIST dataset, and conduct scalability research based on incrementally generated datasets. The results show that the proposed algorithm exhibits higher AUC values, better subspace quality, and improved time efficiency in ultra-high dimensions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

The VAE-FastGA anomaly detection model based on subspace and weakly correlated ultra-high-dimensional data

  • 期刊论文

  • 发表日期：

    2025年2月14日

查看全文
  • Junhang Wan,
  • Yanping Chen,
  • Cong Gao

摘要

The rapid development of technology has led to a continuous increase in data dimensions, and the detection of high-dimensional data growing from tens of dimensions to millions of dimensions. As a result, the negative impact of the dimensional curse on anomaly detection has become increasingly remarkable. Although subspace research has been quite extensive, it has been difficult to extend to ultra-high-dimensional (UHD) datasets. To address this issue, this paper proposes the VAE-FastGA subspace search model, which is based on the idea of feature selection and aims to efficiently screen out anomalous subspaces from weakly correlated UHD data. The model integrates variational autoencoder (VAE) networks with accelerated genetic algorithms (FastGA), using VAE reconstruction to guide the search process of the genetic algorithm. An acceleration mechanism is designed to increase search efficiency by discarding invalid attributes during genetic operations and iteratively preserving anomalous subspaces. To mitigate the effects of sample sparsity, a probabilistic statistical model is introduced to optimize the anomalous subspaces. Finally, the idea of hyperdimensional spheres is integrated to improve the detection of marginal anomalies. The experimental sections organize precision experiments based on the KDD-CUP’99, MNIST, and UCSD datasets, perform subspace evaluations based on the MNIST dataset, and conduct scalability research based on incrementally generated datasets. The results show that the proposed algorithm exhibits higher AUC values, better subspace quality, and improved time efficiency in ultra-high dimensions.