Anomaly detection is a crucial step in ensuring data quality. How to accurately and comprehensively identify outliers in data has always been a core challenge in the field of data science. Although existing studies have proposed a variety of anomaly detection methods for tabular data, some methods are often limited to specific types of data and are difficult to play an effective role in general scenarios. While machine learning-based methods can cover multiple types of errors, they are highly dependent on a large amount of high-quality labeled data, resulting in high system costs. The introduction of few-shot learning techniques has alleviated the labeling burden to a certain extent, but the problem of uneven distribution between labeled samples and unlabeled data still significantly restricts the improvement of model performance. To this end, this paper proposes a novel tabular data anomaly detection method (TP-ADM) from the perspective of data selection, adopting a three-stage learning paradigm to solve the above problems. First, a genetic algorithm is used to complete high-quality sample screening and initial label assignment; second, an improved label diffusion algorithm is introduced to alleviate the problem of uneven data distribution; finally, an iterative adaptive generation strategy for pseudo-labels is designed to achieve comprehensive determination of abnormal data. The experimental results on five benchmark datasets show that TP-ADM achieves excellent detection performance without relying on external knowledge bases. In addition, the data selection module proposed in this paper has a plug-and-play architecture feature, which can be integrated into existing anomaly detection frameworks, and its significant role in improving the performance of mainstream models has been verified in experiments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Anomaly Detection Method for Tabular Data Based on a Three-Phase Learning Paradigm

  • Yufei Li,
  • Hongtao Song,
  • Qilong Han

摘要

Anomaly detection is a crucial step in ensuring data quality. How to accurately and comprehensively identify outliers in data has always been a core challenge in the field of data science. Although existing studies have proposed a variety of anomaly detection methods for tabular data, some methods are often limited to specific types of data and are difficult to play an effective role in general scenarios. While machine learning-based methods can cover multiple types of errors, they are highly dependent on a large amount of high-quality labeled data, resulting in high system costs. The introduction of few-shot learning techniques has alleviated the labeling burden to a certain extent, but the problem of uneven distribution between labeled samples and unlabeled data still significantly restricts the improvement of model performance. To this end, this paper proposes a novel tabular data anomaly detection method (TP-ADM) from the perspective of data selection, adopting a three-stage learning paradigm to solve the above problems. First, a genetic algorithm is used to complete high-quality sample screening and initial label assignment; second, an improved label diffusion algorithm is introduced to alleviate the problem of uneven data distribution; finally, an iterative adaptive generation strategy for pseudo-labels is designed to achieve comprehensive determination of abnormal data. The experimental results on five benchmark datasets show that TP-ADM achieves excellent detection performance without relying on external knowledge bases. In addition, the data selection module proposed in this paper has a plug-and-play architecture feature, which can be integrated into existing anomaly detection frameworks, and its significant role in improving the performance of mainstream models has been verified in experiments.