<p>Video anomaly detection (VAD) plays a critical role in various engineering applications such as safety monitoring and traffic management. Current deep learning-based methodologies for VAD predominantly focus on frame reconstruction and prediction strategies. However, insufficient modeling of long-range spatio-temporal regularities and latent motion transitions in video sequences limits the effectiveness of both paradigms. Inspired by video coding and decoding, we formulate unsupervised video anomaly detection as a restoration task in which intermediate frames are recovered from the start and end frames of an object-centric clip. This formulation requires the model to infer multiple missing frames from sparse temporal observations, thereby encouraging stronger modeling of object-centric spatio-temporal context, including motion continuity, local appearance evolution, and temporal dependency patterns. To support this task, we introduce a Contextual Memory Bank (CMB) with a two-stage memory-coordination learning strategy to store latent motion-context representations from normal sequences and retrieve relevant spatio-temporal priors when only boundary frames are available. We further employ memory query decomposition for local context matching, which improves the consistency between the recalled context and the current input. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed framework for video anomaly detection.</p>