Video anomaly detection plays an increasingly crucial role in intelligent surveillance systems. Inspired by previous unsupervised methods, this paper focuses on detecting frame-level anomalies with long-term temporal dependencies. To this end, we propose a dual-scale temporal dependency learning method for video anomaly detection model, which consists of two main modules: a single-frame reconstruction module and a multi-frame feature enhancement module, processed end-to-end without relying on any pre-trained models. To validate the proposed approach, we introduce a new Elevator dataset containing various types of remote temporal dependency anomalies. Experimental results on the self-constructed Elevator dataset and two benchmarks demonstrate the effectiveness of our proposed approach.