Anomaly Detection (AD) technology has received much attention recently, especially in industrial quality inspection applications. Most existing unsupervised AD methods assume that the training data contains only normal samples, which is difficult to satisfy in practice. When the training data are mixed with even a small number of defective samples, the AD methods, that use distillation learning, will be negatively affected, leading to significant performance drops. To tackle this issue, in this paper, we proposed an approach, namely \(\text {AD}^2\) , to conduct anomaly detection during the training phase of an anomaly detection model. Specifically, we devise a Non-Major Feature Elimination (NMFE) module to eliminate the prominent anomaly-related discrepancy information and adopt an Anomaly Training Data Removal (ATDR) strategy to identify outliers in the training data, preventing abnormal information from affecting model training. During the inference phase, \(\text {AD}^2\) does not introduce any extra computation overhead. Experiments demonstrate that \(\text {AD}^2\) can successfully alleviate the performance deterioration caused by polluted training samples. On the MVTec LOCO dataset, when \(10\%\) of the training set is corrupted by anomalous samples, \(\text {AD}^2\) can significantly improve the image-level AUROC from 0.793 to 0.865 compared to the ordinary AD method, without sacrificing any inference efficiency. \(\text {AD}^2\) provides an effective solution for issues of data uncertainty in anomaly detection. The source code will be released.