Anomaly detection seeks to identify the patterns of instances distinguishable from normal ones. However, current methods primarily align with either one-class or open-set scenarios, leading to an inadequate exploration of anomalous examples. In this paper, we propose a weakly-supervised approach, the Dual Memory-guided Probabilistic Model (DMPM), to explore the comprehensive knowledge of normal and abnormal instances. Employing such dual memory banks, our model provides opposing guidance for probabilistic models during the denoising procedure. We illustrate the effectiveness of our DMPM in addressing weakly-supervised anomaly detection and conduct extensive experiments on popular industrial benchmarks, i.e., MVTec AD and VisA. Moreover, we highlight the adaptability of the unified DMPM, demonstrating its compatibility with diffusion-based approaches that perform on image or latent space.