Image emotion analysis is attracting more and more attention as versatile information is delivered on social media. Due to the ambiguity and subjectivity of emotion, learning emotion distribution in images is more challenging than classifying emotions in images. For instance, Contrastive Language-Image Pre-training (CLIP) based methods perform well in emotion classification, but may cause generalization deterioration problem when it comes to image emotion distribution learning (IEDL). To cope with this issue, a CLIP-based method is proposed in this paper, namely Emotion-Aware Semantic Constraint and Correlation Refinement (ESCOR). Specifically, descriptive texts containing object semantics are incorporated with dominant emotions to guide visual representation learning, thereby reducing the loss of semantic information. In addition, a new module named emotion correlation refinement is designed to facilitate CLIP to learn class representations enriched with correlation priors. Extensive experiments on public datasets demonstrate that ESCOR outperforms state-of-the-art methods for IEDL.