Fine-Grained Visual Classification (FGVC) is a computer vision task that involves classifying subtle differences in images. While the Vision Transformer (ViT) is excellent at capturing long-range dependencies in sequences, it may not be as effective in extracting local image features that are crucial for fine-grained tasks. To address this issue, we propose a multi-scale multi-attention transformer model (MMAT), which consists of two parts. Firstly, a method for fine-grained feature fusion is used to aggregate and compress feature maps from the last ViT layer, which serves as input for the classifier. We evaluated various pooling strategies to determine their impact on classification accuracy. Secondly, an attention-based Local Feature Extraction Module leverages attention weights from ViT’s multi-head self-attention mechanism to identify key local regions for further feature extraction and classification via part-CNNs. Experiments were conducted on the public dataset CUB-200-2011, as well as a self-built wild fungi dataset. The results of the experiments demonstrate the model’s superior performance and wide applicability in the field of FGVC.