Machine learning models suffer from serious performance declination when facing out-of-distribution datasets. In recent years, numerous researches on domain generalization (DG) have been made to address that issue and improve the model’s generalization. With the development of multimodal models, there are an increasing number of works considering utilizing large vision-language models to achieve DG. In this paper, we proposed Semantic Information Extraction and Target Alignment (SIETA) to take alignment both in the training and testing phases. We choose pre-trained CLIP as the teacher to guide our model to learn the ability to extract semantic information by minimizing the distance between the representations of CLIP’s text encoder and those of our encoder during training and use knowledge distillation to transfer the abundant prior knowledge of CLIP into our model. In the inference phase, we leverage Test-Time Adaptation (TTA) to slightly align our model with the target domain to further enhance the model’s generalization. We conducted experiments on four DG benchmark datasets, and the results show that our method significantly improves the model’s generalization with a smaller size than CLIP and is versatile to combine with other DG methods.