Recent foundational models have unlocked numerous possibilities for computer-assisted interventions. A critical advancement in this field is the precise estimation of dense relative depth on surgical videos, essential for understanding the 3D positioning of surgical instruments and measuring anatomical structures. However, existing methods often struggle to estimate depth maps that are coherent and smooth over time, leading to noisy and temporally inconsistent depth predictions. We propose TAN, a novel Temporal Adapter Network for monocular depth estimation that enhances the foundational model Depth Anything V2 from image-based to temporally aware depth estimation. Specifically, we design a lightweight temporal adapter and integrate it into the decoder to capture temporal features from consecutive frames. Additionally, we introduce a self-supervised temporal regularization loss, utilizing optical flow to enforce stable depth estimation between consecutive frames. Our experiments, conducted on the SCARED and EndoNeRF datasets, two established benchmarks for evaluating depth estimation models in the surgical domain, demonstrate that the proposed TAN improves both temporal consistency and depth accuracy, achieving at least a 14.29% reduction in OPW and 3.6% in RMSE on SCARED, and 6.2% in OPW and 3.26% in RMSE on EndoNeRF compared to state-of-the-art methods, while running at 97 FPS, making it well-suited for real-time surgical applications.