Federated Learning (FL) enables collaborative model training across distributed devices while preserving data locality. However, privacy vulnerabilities persist, as adversaries may reconstruct sensitive data through gradient inversion attacks during model updates. Although Homomorphic Encryption (HE) can mitigate such risks, its prohibitive computational and communication overhead severely hinders practical adoption, particularly for large-scale models. This paper introduces FedSPE, an efficient FL system featuring Selective Parameter Encryption (SPE) and adaptive privacy mechanisms. By dynamically adjusting encryption thresholds based on data distribution patterns and selectively protecting sensitive parameters, FedSPE reduces HE overhead. We further integrate differential privacy to establish a hybrid defense against gradient attacks, achieving customizable privacy-utility tradeoffs. Extensive experiments demonstrate remarkable efficiency gains: Our innovations in dynamic parameter selection and multi-layered privacy preservation bridge the gap between theoretical security and practical FL deployment, offering a scalable solution for privacy-sensitive applications like edge AI.