Multi-class anomaly detection has been widely explored which trains a unified model for different object categories. However, in practical applications, these models suffer from catastrophic forgetting since data of different categories usually arrives incrementally, and the historical data is unavailable due to privacy constraints. In this paper, we propose a novel approach namely Adaptive distillation and Dynamic neighbor masked Attention for Continual Anomaly Detection (Ada-CAD). To address catastrophic forgetting, teacher-student distillation is employed to retain knowledge from the previous teacher model while learning data of new categories. We propose a pooling and adaptive distillation strategy to balance knowledge retention and adaptation. Additionally, to overcome the identity shortcut problem in traditional Transformer-based architectures, we incorporate a dynamic neighbor masked attention (DNMA) mechanism, which adaptively adjusts masking ranges based on the input features to improve feature reconstruction. Experiments on the widely-used MVTec-AD dataset show that our method achieves superior performance compared to existing state-of-the-art approaches in continual anomaly detection.