Sleep stage classification is a significant measure in assessing sleep quality and diagnosing sleep disorders. Many researchers have investigated automatic sleep stage classification methods and achieved promising results. These methods ignored the neighborhood information fusion of the multi-scaled temporal and spectral features among multiple-channel sleep monitoring signals, resulting in the inability to capture complex spatio-temporal dependencies. In this study, we propose an A Multi-Scale Temporal-Spectral Fusion Network with Self-Supervision Learning, named SSL-MSTFNet, for sleep stage classification. Specifically, A self-supervised learning without negative samples, multi-scaled CNN module and channel-wise attention module are used to extract multi-scaled temporal features and stack the multi-scaled feature maps and fuse them with a channel-wise attention module. Simultaneously, converting signals into spectrograms using a short time Fourier transform (STFT), then VGG-16 network, convolutional block attention module (CBAM) are employed to capture the spectral features from sleep monitoring signals. Finally, we fuse these two features to classify sleep stages into five categories. Experiments are performed on two public datasets: ISRUC-S1 and ISRUC-S3, the SSL-MSTFNet achieves competitive results and is superior to its state-of-the-art counterparts on most of performance metrics.