Pretrained large language models (PLMs) have revolutionized artificial intelligence through their advanced language capabilities, which are further enhanced via fine-tuning for specialized downstream applications. Federated learning (FL) offers a promising framework for distributed fine-tuning of PLMs, preserving data privacy by avoiding raw data sharing and enabling collaborative model refinement. However, the practical deployment of federated tuning for PLMs faces two critical challenges: 1) high computational, communication, and memory costs, particularly on resource-scarce terminals, and 2) indirect privacy leakage risks from gradient or parameter updates. Existing solutions either employ parameter-efficient methods (e.g., LoRA, Adapter) to reduce resource overhead or utilize cryptographic tools or differential privacy (DP) to mitigate privacy risks, treating each challenge in isolation without offering a cohesive approach to address both simultaneously. This paper introduces FedDPZO, a novel FL framework that integrates zeroth-order optimization for gradient estimation with DP-based stochastic perturbations, enabling efficient and privacy-preserving PLM fine-tuning. In contrast to previous work, FedDPZO is explicitly tailored for federated settings and introduces the first unified framework to tackle both efficiency and privacy in federated fine-tuning for PLMs, supported by both theoretical guarantees and empirical validation. Theoretically, we establish a convergence bound for loss reduction per aggregation round, logarithmically dependent on model dimensions, and derive a global convergence rate of \( \mathcal O(r^2 \cdot T^{-1}) \) . Empirically, FedDPZO outperforms DP-based FedAvg by 2.8% on SST-2, 2.6% on BoolQ, and 1.9% on Fed-Dolly, demonstrating its superior effectiveness.