Accurate time series forecasting demands effective integration of temporal dynamics and contextual semantics. While existing attention mechanisms capture numerical patterns effectively, they often neglect domain-specific temporal knowledge. We propose MixRecLGB, a novel framework that synergizes LightGBM with a language-enhanced mixed attention mechanism. Our key contributions include: 1) A recursive VAE architecture (RecLGB) that compresses long historical sequences into hierarchical memory features through progressive latent space learning; 2) A temporal-semantic fusion mechanism that injects frozen language model embeddings into both static linear attention and dynamic self-attention components, preserving temporal order while incorporating contextual knowledge; 3) A parameter-efficient integration strategy that enhances attention computation through adaptive bias injection and feature fusion, requiring minimal architectural modifications. Evaluated on five real-world datasets, MixRecLGB reduces forecasting errors while maintaining computational efficiency. This work establishes a new paradigm for combining deep temporal modeling with efficient gradient boosting, particularly effective for long-term forecasting scenarios.