Federated Learning (FL) employs a distributed training paradigm that eliminates the need for raw data sharing, thereby reducing the risk of privacy leakage. This approach has made FL a prominent research area in both academic and industrial communities. However, several issues persist in FL, such as attackers can infer users’ private data from the local model gradients uploaded by users, and the participation of users with low-quality data can adversely affect the performance of the federated model training. To address these two problems: (1) protecting user gradients to prevent privacy leakage and (2) mitigating the interference of low-quality data. This paper proposes a lightweight Privacy-Preserving Federated Learning Scheme with Robustness Against Low-Quality Users in Cloud-Edge Collaborative Environments. Specifically, we design a lightweight privacy-preserving mechanism that combines the Diffie-Hellman Key Exchange Protocol and masking techniques to protect user gradients and prevent privacy leakage. Additionally, we propose a Hamming-cosine hybrid weighting algorithm to mitigate the adverse impact of low-quality data users. Moreover, our scheme inherently exhibits robustness against user dropouts. Detailed analysis and experimental results demonstrate the security and efficiency of the proposed scheme.