People could use various types of subtle metaphorical language forms to express individual opinions or negative emotion in daily life, such as sarcasm, satire or irony. Sarcasm employs contradictory and incongruous elements to convey the difference between reality and expectation. Detecting precisely whether comments, opinions or conversations have sarcastic intention is crucial for understanding the talkers’ feeling and attitude. Since the complexity of sarcasm requires certain contextual information to catch on ironic meaning correctly, how to leverage multimodal information to grasp sarcastic remarks has become a hot research topic. A Multimodal Sarcasm Detection method based on Contrastive Learning (MSDCL) is proposed in this paper to exploit multimodal information and incongruity knowledge for improving performance. MSDCL extracts fine-grained features from text and image and captures contradictory semantics between text and image by using the multi-head self-attention mechanism. A supervised contrastive learning is implemented to better learn intra- and inter-class relationships among sarcastic and non-sarcastic data, where the classification loss and contrastive learning loss are integrated to guide the multimodal data embedding. The experiment results show MSDCL outperforms the compared methods.