Vision-Language Models (VLMs) align different modalities within a shared space through joint training, thereby enabling powerful zero-shot inference and generalization capabilities. However, existing models still struggle with fine-grained semantic alignment. Recent approaches employ visual augmentation techniques (e.g., random cropping) to incorporate multi-view information, but the generated image regions often lack semantic consistency, hindering precise alignment with textual concepts. To address this, we propose a novel method termed Concept-based Semantic Alignment (CSA) that introduces a dual-branch architecture to extract concept-level representations from both visual and textual modalities. Specifically, CSA identifies high-attention visual regions and performs region-level cropping to focus on semantically salient content. Simultaneously, we leverage Large Language Models (LLMs) to generate concept-level semantic descriptions and enhance fine-grained cross-modal alignment. Additionally, we design a multi-granularity bidirectional alignment strategy to further refine cross-modal alignment at the concept level. Experimental results demonstrate that CSA achieves improved performance on both zero-shot and few-shot classification tasks.