Nowadays, vast amounts of unlabeled data are readily available. Obtaining the labels requires domain experts to manually revise and annotate the cases in an expensive and time-consuming process. Active learning addresses this issue. However, traditional active learning methods disregard the class imbalance problem. In this paper, we propose Adaptive Resampling and Active Classification for Thresholded Anomalies (A-REACT) Algorithm, a novel method for active learning that tackles the class imbalance while preventing the introduction of noisy synthetic samples. A-REACT incorporates outlier detection in the active learning procedure to select the best samples to be labeled by experts. An adaptive resampling method is used to ensure the reduction of noisy synthetic cases while improving the execution time. An extensive experimental study on 8 datasets shows the clear advantage in terms of performance of A-REACT algorithm when compared to the state-of-the-art solution on three different performance metrics. Moreover, A-REACT also exhibits lower execution times than the alternative methods.