Visual emotion recognition (VER), which aims to understand human emotional reactions to different visual stimuli, has garnered increasing attention. Due to emotions’ inherent subjectivity and complexity, annotating a reliable dataset is time-consuming. To reduce dependence on labeled data, unsupervised domain adaptation (UDA) aims to adapt a model trained on a well-labeled source domain to the unlabeled target domain. Recently, large-scale pre-trained vision-language models (VLMs) such as CLIP have demonstrated noteworthy zero-shot capability, performing excellently on conventional UDA tasks. However, existing methods encounter specific challenges with abstract tasks like VER, particularly due to the misalignment between CLIP and emotion spaces, as well as the affective gap between domains, leading to performance decrease. In this work, we propose EIDA, a CLIP-based method designed for emotional unsupervised domain adaptation. To address the misalignment between CLIP and emotion spaces, we propose explicit and implicit interactive emotional spaces: the implicit emotional space removes redundant semantic information from text embeddings to generate pseudo-labels, while the explicit emotional space directly maps visual semantics to emotion categories, constructing a distinct space for abstract emotion features. To address the affective gap between domains, we project source and target domains into a shared emotional subspace to align category prototypes. Extensive experiments demonstrate the effectiveness of our proposed EIDA, showing 38.99% and 4.64% improvements over state-of-the-art emotional UDA and CLIP-based UDA methods on the FI dataset, respectively.