Multimodal sarcasm analysis has received considerable attention in the artificial research area. Compared to traditional sentiments, sarcasm is more complex inherently, as its diverse expression and rich hidden implications, which require more comprehensive cues from different dimensions for sarcasm analysis. However, existing works only attend to the polarity-aware cross-modality sentiment incongruity or the semantic-aware sarcasm diversity, resulting in the fuzzy sarcasm analysis. In this paper, we propose a multimodal polarity-semantic coupling network (MPCN) to simultaneously deal with the polarity-aware and semantic-aware fine-grained multimodal sarcasm analysis. Specifically, the sentiment polarity factors module is proposed to exploit the positive and negative polarity factors among multiple modalities. Additionally, the semantic factors module is presented to facilitate the intrinsic private sarcasm properties. Then, the obtained polarity and semantic factors are further integrated into the coupling sarcasm representation. The proposed coupled learning mechanism indeed provides us with a novel analysis direction for the multimodal sarcasm analysis task. Various experiments demonstrate that the proposed manner achieves better performance compared to the baselines.