Learning-based anomaly detection methods train models using normal data samples to capture normal behavioral patterns for identifying anomalies. Unfortunately, the distribution of normal data in Internet of Things (IoT) environments always shifts because of device upgrades and security patch implementations. Although few efforts have been made, current approaches fail to effectively exclude anomalies during the phase of normality shifts detection, and suffer from a high labeling costs during the adaptation phase. To overcome these drawbacks, this study proposes a comprehensive solution encompassing three modules. The first Normality Shift Filter is preposed to filter anomalies, meanwhile, leverages the latent space representation of an Autoencoder (AE) to capture representative samples, thereby significantly reducing the cost of labeling. The second Normality Shift Detector employs a tripartite ensemble method to accurately detect normality shifts. The third Normality Shift Adapter is designed as a customized progressive neural network to further adapt incoming shifts while retaining the valuable knowledge learned from historical data. Empirical tests conducted on open datasets demonstrate that our proposed method outperforms the state-of-the-art (SOTA) methods.