Vision Transformers (ViTs) have recently demonstrated excellent performance in various computer vision tasks, but the substantial computational costs pose challenges for practical deployment. Since the computational complexity of transformer models scales quadratically with the number of image tokens, and not all tokens contribute to the final prediction, reducing the number of input tokens has become a prevailing paradigm for accelerating ViTs. Token fusion is an effective method to reduce redundant tokens. However, some existing works simply average token features for fusion, which may result in feature dispersion. Additionally, fusing certain informative tokens may have a negative impact on image recognition. To this end, we propose a new token merging method and token retention strategy. Specifically, we first divide the tokens into two roughly equal-sized groups and perform similarity-based weighted fusion on the most similar token pairs. Then, we apply attention head importance to weight the class attention scores, and retain the top-k important tokens from the fused tokens. We conducted extensive experiments on ImageNet classification tasks with ViTs to validate the effectiveness of our method.