Wi-Fi-based gesture recognition, driven by deep learning, holds significant promise for privacy-preserving and all-weather sensing. However, current methods typically rely on large amounts of labeled data, and Wi-Fi signals vary significantly across gestures, leading to severe performance degradation when models encounter unseen gestures. To address these challenges, we explore the potential of transferring knowledge from large pre-trained language models to improve the generalization of Wi-Fi-based gesture recognition systems. To this end, we propose a zero-shot gesture recognition framework, named Wi-CLIP. Inspired by the vision-language pre-training model CLIP, our method constructs a cross-modal radio frequency-text model centered on aligning Wi-Fi signals with textual semantics. Specifically, we develop a novel Wi-Fi signal encoder and a BERT-based text encoder, aligning the two modalities within a shared semantic space using contrastive learning. Our framework achieves an average recognition accuracy of 89.12% across 6 gestures. Notably, when trained on only 5 gestures, Wi-CLIP demonstrates a remarkable zero-shot recognition accuracy of 78.79% on the sixth, previously unseen gesture. This highlights its strong generalization capability and effectiveness in cross-modal representation learning.