As an emerging decentralized machine learning paradigm, federated learning (FL) facilitates collaborative model training while preserving data privacy and security. However, due to its distributed nature, byzantine attackers can significantly compromise the global model through malicious update injection. Existing defense strategies typically rely on anomaly detection of the received local model updates from the perspective of server. This study reveals security vulnerabilities in current detection approaches when confronting hybrid byzantine attacks employing deception mechanism. To address this challenge, we shift the defensive perspective from global anomaly detection to client perspective analysis by leveraging the similarity among client model updates. We propose a dual-voting mechanism for robust federated aggregation, which strategically selects trustworthy updates through similarity-based voting in each training round. Extensive experiments demonstrate the effectiveness of our approach in mitigating sophisticated byzantine attacks.