Recent developments in language modeling have markedly enhanced the capabilities of natural language processing systems. Nevertheless, these models encounter difficulties in continual learning (CL), especially in preserving previously learned knowledge when integrating new data—an issue commonly referred to as catastrophic forgetting. We revisited the concept of data replay in continual learning and introduced two novel improvements: the Influence-Guided Sampling (IGS) strategy for memory buffer construction and the Contrastive Gradient Modulation (CGM) mechanism for parameter update, aiming to mitigate catastrophic forgetting and enhance knowledge transfer. IGS-CGM not only replays past data but also modulates the current task’s gradient through a contrastive analysis with gradients from previous tasks, thereby preserving the model’s proficiency in previously acquired domains while learning new ones. We conducted extensive experiments on three CL benchmarks, covering traditional finetuning and instruction finetuning for large language models, demonstrating its effectiveness in mitigating catastrophic forgetting and enhancing knowledge transfer.