Contrastive learning is a mainstream method of self-supervised learning, which has achieved excellent performance recently. It refers to learning representative features by discriminating the similarities and dissimilarities between images without labels. However, contrastive learning focuses more on the deep semantic information, it falls short in terms of spatial sensitivity. Besides, neural networks with different depths extract different information, all of which is critical for models to perform decision-making. Nevertheless, traditional contrastive learning methods do not sufficiently utilize the input features. To address these issues, we propose SGMNet, which adopts a cross-space mutual optimization structure combining contrastive learning with position prediction task. Moreover, to improve the utilization of information, SGMNet integrates features from different levels so that it can obtain texture and semantic information. In addition, considering the high redundancy of shallow feature, in order to make the model focus on the important parts and avoid it learning some meaningless “shortcuts”, we design a feature fusion module to extract the information in the shallow feature map with deep feature map, which enables the model to focus on the semantic-related important detail features. Experimental results show that SGMNet gives competitive performance on multiple datasets.