Knowledge-aware recommendation aims to enhance recommendation performance by leveraging item-side information in knowledge graphs (KGs). Existing methods typically assign identifiers (IDs) to items and entities in a KG and model their structural relations. However, these methods overlook rich semantic information and therefore fail to capture the latent semantic associations between users and items, leading to suboptimal performance. To address this, we propose a Semantic-empowered Knowledge-aware Recommendation approach called SeeKRec, which captures the semantic-collaborative features of users and items in an extract-refine-integrate manner. Specifically, to extract semantic information, we design an adaptive semantic preference extractor that hierarchically integrates intra-user preference discovery and inter-user semantic alignment, achieving accurate preference understanding and uncovering latent cross-user relations. To achieve task-driven semantic refinement, we design a sample-then-enhance framework that first samples task-relevant semantics through meticulously designed semantic sampling and then enhances them via a task-aware semantic calibrator, yielding high-quality semantic representations aligned with recommendation objectives. To seamlessly integrate refined semantic representations with conventional collaborative signals, we propose a semantics-aware mixture-of-experts model with a distill-then-fuse strategy, which distills semantics to bridge the dimensional gap, and fuses them with ID embeddings via dynamic expert routing to capture diverse semantic-collaborative relations. Extensive experiments show SeekRec achieves state-of-the-art results.