Speaker verification is to judge the similarity between two unknown voices in an open set. In recent years, speaker verification models based on Transformer and CNN networks have primarily focused on either global feature information or local feature granularity, rather than considering both together. To address the limitation, we propose a novel model called Voice Transformer (VOT), which integrates parallel transformers at multiple scales. A deep feedforward sequential memory network (DFSMN) is incorporated into the attention part of these transformers to increase local feature granularity. The attentive statistics pooling layer is employed to focus on important frames and form global utterance-level features. Considering the similarity of speech characteristics, we propose Additive Angular Margin Focal Loss (AAMF) to effectively tackle the challenge of hard samples. We evaluated the proposed approach on the VoxCeleb1 and CN-Celeb2 datasets, demonstrating that VOT surpasses most mainstream models. The code is available at https://github.com/luckyerr/Voice-Transformer_Speaker-Verification .