Pre-trained language models(PLMs) suffer from catastrophic forgetting in continual learning, as sequential task training overwrites previously learned representations. The model’s ability to remain old tasks is referred to as stability, while its adaptability to new tasks is called plasticity. Therefore, the key to addressing this challenge requires balancing model plasticity with stability. To address this issue, in this paper, we propose a novel method to achieve a balance between model stability and plasticity, thereby mitigating catastrophic forgetting. More specific, our proposed approach leverages parameter isolation and subsequent combination strategy. Initially, in training stage, the model adapts on each downstream task via parameter isolation method to prevent potential inference among different tasks. We then combine all trained parameters which containing acquired knowledge by model merging method and finally apply to the backbone model. Empirical evaluations on continual language learning benchmarks substantiate the effectiveness of our approach, revealing a marked enhancement over existing state-of-the-art approaches.