Unrestricted adversarial attacks have proven effective in fooling DNNs by adding subtle, unconstrained perturbations to input data. However, previous methods rely on random noise injection to generate unrestricted adversarial examples (UAEs) without semantic guidance, resulting in low black-box transferability against unseen models. To address this issue, we introduce PDAttack, a novel framework that utilizes prompt-driven semantic expansion to bolster adversarial transferability. First, we leverage a Visual Language Model (VLM) to craft perturbation prompts, thereby expanding the semantic space of UAEs. In other words, this means that the perturbations can act on features with different semantic associations. Second, two deception strategies are implemented in the latent space of the diffusion model: (1) directly perturbing the image to deceive the classifier, and (2) indirectly perturbing the generation of UAEs by embedding prompts into the denoising process of Stable Diffusion Model (SDM). Unlike prior works, PDAttack integrates a Cross-Iteration Fusion scheme during the diffusion process to force different labels to have similar distributions, thereby blurring the semantic boundaries between different labels. Concluding our framework, we introduce a hierarchical self-attention optimization module that simultaneously enhances computational throughput and maintains the visual imperceptibility of the adversarial examples. Compared with existing methods, the proposed PDAttack and its enhanced version PDAttack-X achieve an average of 73.9% (about +9%) ASR and 81.9% (about +16%) ASR in black-box attack against four prevalent DNNs with two different architectures on ImageNet-compatible dataset. Code is available at https://anonymous.4open.science/r/PDAttack .