Temporal action detection (TAD) faces significant bottlenecks in scalability and efficiency due to its heavy reliance on costly video annotations and excessive computation. Although vision-language models (VLMs) provide promising alternatives for visual tasks, their application to TAD reveals a critical cross-modal semantic gap. This gap stems from two limitations. One is the semantic granularity discrepancy, where atomic action labels are insufficient to capture dynamic spatiotemporal processes, causing misalignment with VLM embeddings. The other is the temporal modeling incapacity, as image-pretrained VLMs lack understanding of motion dynamics. To bridge this gap, we propose a novel framework through two key innovations. First, our concept enrichment method leverages large language models (LLMs) to decompose abstract action annotations into concrete concepts, generating semantically aligned textual representations. Second, our temporal-aware adaptation method distills motion dynamics across video frames, enabling VLMs to interpret temporal evolution. The synergy of these components establishes precise vision-language correspondence. Extensive experiments on THUMOS14 and ActivityNet-1.3 benchmarks demonstrate that our method achieved a lead of 2.9% and 0.5% respectively, confirming its efficacy in establishing cross-modal semantic alignment and advancing VLM-based temporal action detection.