Multimodal metaphors are prevalent in social media. Their unusual and creative usage makes multimodal metaphor detection exceptionally challenging. Existing research on multimodal metaphor detection overlooks the intention and semantic incongruity inherent in metaphors. The intention-semantic incongruity triggered by these conflicts is the crucial cue for detecting multimodal metaphors. To address this gap, we propose the Intention-Semantic Incongruity Perception Network (ISIPN) for multimodal metaphor detection. On the intention level, we utilize the siamese network to perceive cross-modal intention incongruity. Subsequently, we implement the continuous contrastive learning strategy to enhance intention embeddings. On the semantic level, we construct multimodal semantic graphs and integrate multi-head graph attention networks with neighbor contrastive learning to perceive cross-modal semantic incongruity for metaphor detection. Experimental results demonstrate the effectiveness and superiority of ISIPN, surpassing current state-of-the-art methods and multimodal large language models.