Multimodal Entity Linking (MEL) aims to utilize complementary modalities to supplement textual information, facilitating the linking of mentions to the Knowledge Graph (KG). Existing methods mostly focus on the static fusion of multimodal features to obtain mention representations while neglecting the varying contribution of complementary modalities across samples and overlooking the relevance between candidate entities and mentions. To address these issues, we propose an Adaptive Semantic-Aware Fusion method for multimodal entity linking (ASAF), which mainly contains two components: the Adaptive Multimodal Fusion (AMF) module and the Gated Cross-Attention (GCA) module. Specifically, we first introduce an AMF module using Kullback-Leibler (KL) divergence to dynamically assess the contribution of visual modality based on the semantic distribution similarity between the textual and visual modalities for each sample. This ensures the adaptive adjustment of information for multimodal fusion, reducing the introduction of noise or redundant information. Additionally, two GCA modules are introduced to capture the semantic correlations between text and image, as well as between entities and mention, respectively. Experiments conducted on three public datasets demonstrate that ASAF outperforms baselines, validating the effectiveness of the proposed approach.