Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) aims to retrieve natural images of the same category using a sketch as a query in zero-shot scenarios. Benefiting from the outstanding content understanding, CLIP-based ZS-SBIR methods have achieved unprecedented retrieval performance. These methods typically use semantic or various auxiliary information captured by CLIP to align the feature spaces of sketch and photo. However, they often overlook the differences between different visual modalities, leading the model to produce compromised retrieval results. To address this issue, we propose a novel Cross-modality Coupled Prompt Learning (CCPL) method for ZS-SBIR. Specifically, we first introduce modality-specific prompts during the encoding stage to capture the distinct features of each modality. In this way, the unique features of each modality can be preserved, thereby enhancing the overall understanding of model for visual information from different modalities. Then, we design a coupling block to model the inter-stage relationships of features, ensuring the encoder bridges the subtle visual differences between the two modalities. Finally, we develop a double anchor triplet loss with adaptive margin to balance discrimination and generalization, while employing KL divergence to align the sketch-photo similarity distributions between the learned and existing knowledge. Extensive experiment results have demonstrated our significant performance improvements on three datasets.