Feature matching across images is critical for computer vision tasks such as visual localization. Previous methods relied on extracting a large amount of redundant local reliable features, which limits efficiency and accuracy when handling challenging large-scale environments. We propose a fast feature matching method that incorporates high-level semantic information. The previous detector-free LoFTR method performs well in texture-scarce scenarios but still faces accuracy and efficiency limitations. We re-examine its design and propose improvements to enhance both performance aspects. Specifically, the reliability of matching is enhanced by implicitly embedding semantic information within the feature matching process. Additionally, our analysis of LoFTR revealed that performing Transformer operations across the entire feature map can be redundant since local information is shared. Therefore, we employ an aggregated attention mechanism to improve the efficiency of feature learning. Given that spatial variance in LoFTR’s refinement module hinders matching precision, we introduce a two-stage correlation layer to achieve accurate sub-pixel matching. Experiments show that our model demonstrates significant performance advantages compared to LoFTR and other advanced matching algorithms.