Detecting depression poses a significant challenge in mental health assessment due to the complexity and subtlety inherent in behavioral manifestations. While traditional diagnostic methods rely on clinical interviews and questionnaires, they are often subjective, time-consuming, and may miss subtle behavioral indicators. The development of automated depression detection systems faces two critical challenges: effectively modeling the complex temporal patterns in behavioral manifestations, and capturing the intricate relationships between audio and visual modalities. To address these challenges, we propose MTCA-ViT, a novel multi-modal temporal contrastive vision transformer framework for automated depression detection. Our approach introduces two key innovations: a Cross-modal Contrastive Learning (CCL) strategy that effectively aligns representations while preserving modality-specific characteristics, and a Temporal Adaptive Attention (TAA) mechanism that dynamically captures depression-relevant temporal patterns in both modalities. Extensive experiments on the LMVD dataset demonstrate that our model outperforms existing methods across multiple evaluation metrics. This work represents a significant advancement in automated depression detection, offering both theoretical contributions to deep learning and practical value for mental health assessment.