Sleep stage detection is an important aspect of assessing sleep quality. Conventional research has primarily focused on automatic sleep scoring using single-channel electroencephalogram (EEG) data. However, accurately detecting sleep stages based solely on EEG signals is challenging due to artifacts and similarities in EEG patterns introduced by eye movements during certain sleep stages. To address these issues, we propose a multi-modal framework that combines contrastive learning and sequential encoding to enhance sleep stage detection by leveraging both EEG and electrooculogram (EOG) signals. Our framework consists of several key components, including contrastive learning to optimize convolutional encoders for epoch-level feature representation, two distinct sequence encoders to capture temporal dependencies at the sequence level, and an ensemble prediction strategy to improve sleep stage detection performance further. Through extensive comparisons with state-of-the-art techniques on three publicly available datasets, our proposed algorithm consistently demonstrated superior performance across various evaluation metrics, thereby validating its effectiveness and potential to outperform existing methods in automatic sleep stage detection.