Federated learning (FL), as a promising paradigm for collaborative model training across distributed devices, faces susceptibility to Byzantine attacks. These attacks compromise the convergence and integrity of the global model by having Byzantine attackers periodically injecting crafted and malicious model updates to the learning process. Existing secure aggregation schemes adhere to the typical FedAvg algorithm and require that malicious updates exhibit discernible deviations from benign ones, which, however, is unrealistic in practice because of non-i.i.d. data. In this paper, we propose a Byzantine-robust FL system from the perspectives of both innovative secure aggregation and algorithm design. At first, we introduce a novel diverse aggregation technique aimed at approximating the average of model updates of benign clients, thereby enhancing its resilience against various types of Byzantine attacks. Then, we propose the FedGALVR algorithm, which leverages the techniques of local variance reduction and global adaptive optimization to alleviate the divergence among local models caused by non-i.i.d. data, thus further enhancing the Byzantine robustness. Finally, we substantiate the effectiveness of our proposed approach through extensive experimental validation, demonstrating its superiority over the counterparts. Our work sheds light on the critical challenges posed by Byzantine attacks and provides practical solutions to bolster the security and reliability of FL systems.