Federated learning (FL) is a novel distributed machine learning paradigm where multiple users collaborate to train a global model under the coordination of a central server. However, some prior researches in the domain of FL do not adequately consider adversarial actions. Moreover, these studies often entail significant computational and communicative expenditures, and frequently fall short in achieving an optimal equilibrium between efficiency and privacy preservation. Such limitations is impeding the practical application of FL systems. In this paper, we propose PBFL, building a bridge between privacy and robustness while balancing efficiency. Compared to previous works, PBFL boasts stronger privacy protection and lower computational and communicational overhead. Specifically, on the privacy side, we use Multiparty homomorphic encryption (MHE) as a privacy building block (secure against post-quantum attacks), and in terms of robustness, we build a Byzantine robust aggregation scheme on ciphertext based on the similarities between benign user parameters. We provide a detailed theoretical analysis to demonstrate the security and convergence properties of the scheme. The experiment on two image classification datasets shows that the aggregation weight of Byzantine user is reduced by at least 99.88% compared to benign users, which is negligible in aggregation.