With the growing pervasiveness of encryption techniques, the detection of encrypted malicious traffic presents a significant challenge to cybersecurity. Traditional attack signature-based detection methods struggle to cope with the intricacies of encrypted traffic, while most of existing machine learning-based methods heavily rely on numerous labeled samples. Besides, these methods often perform poorly in real-world scenarios where there is a substantial amount of unknown malicious encrypted traffic. In this paper, we propose MEMO, a novel unknown malicious encrypted traffic detection approach integrated with metric learning and an order-aware pre-training framework. MEMO leverages the abundant supply of unlabeled encrypted traffic for pre-training purposes and implements a unique burst order ranking task to effectively learn coherent traffic patterns. Furthermore, MEMO employs a metric learning fine-tuning strategy to make the decision boundaries between known and unknown malicious traffic more distinct. The experimental results on two real-world datasets show that MEMO significantly outperforms state-of-the-art methods, achieving up to 95.65% accuracy and exhibiting robust performance in few-shot learning scenarios.