Line art colorization constitutes a pivotal step in the production of hand-drawn animation. In consecutive frame scenarios, prevailing approaches predominantly depend on optical flow and implicit feature matching for color propagation, which frequently results in inaccuracies and instability owing to the lack of explicit semantic constraints. To address this limitation, we propose the Progressive Geometric-to-Semantic Fusion Network (PGSFNet). The framework introduces semantic information explicitly through a two-stage progressive strategy: a Geometric Structure Preservation Stage, where RAFT-based optical flow, combined with the proposed Pixel-Window Self-Attention (PWSA), achieves pixel-level geometric alignment between reference and target frames; and a Semantic Refinement Stage, where the proposed Segment Token Cross-Attention (STCA) operates on joint representations derived from a UNet backbone and CLIP embeddings, thereby establishing robust semantic correspondences across segmentation regions. This progressive design effectively alleviates the limitations of purely implicit geometric alignment and provides enhanced semantic interpretability for the colorization process. Experimental results demonstrate that PGSFNet improves color accuracy (Acc) by 2.01% over baseline methods and effectively mitigates color bleeding, semantic mismatches, and detail omissions.