Transferring contrastive image-text pre-trained models, such as CLIP, to the video domain for action recognition has received much attention due to its competitive performance, and some representative methods have emerged. However, while these methods have achieved some success, they tend to learn the features of video and text independently, ignoring the correlation between them. Furthermore, there is still room for further improvement in using temporal information and designing the text prompt. To address these issues, we propose a correlation-guided image-to-video transfer learning model (CGI2VTL) in this paper for effective and efficient video recognition. Specifically, the network architecture we proposed consists of five key components: a CLIP visual/text encoder for generating image representation and text representation; a visual/text adapter for fine-tuning the backbone; a temporal encoder module for extracting temporal information; a class-specific text prompt learning scheme for modeling contextual words; and a canonical correlation analysis (CCA) module to better correlate video and text information. Our extensive experimental results on three well-known action recognition datasets (HMDB-51, UCF-101, and SSv2) have demonstrated the effectiveness of the proposed model.