Federated learning (FL) is a privacy-enhancing distributed machine learning framework, but attackers can reconstruct the client’s private data from the exchanged model updates by launching a gradient inversion attack (GIA). Current defense methods struggle to strike an optimal balance between privacy protection and performance. In this paper, we propose a defense method driven by cost-effectiveness parameters, designed to safeguard privacy while maintaining the utility of the global model. This strategy increases the difficulty of data reconstruction by restricting the upload of parameters with high privacy leakage risks, while preserving knowledge-rich parameters to maintain model performance, achieving a better balance between privacy and utility. To validate the effectiveness of our method, we conduct experiments in both Computer Vision (CV) and natural language processing (NLP) tasks. The results show that our approach can effectively protect users’ image and text privacy while preserving model performance as much as possible.