The use of Chain of Thought (CoT) distillation to enhance the reasoning ability of small pretained language models (SPLMs) from large language models (LLMs) has been widely studied. However, during the distillation process, the parameters of pre-trained small models could be easily modified arbitrarily or randomly, leading to the loss of inherent knowledge in the SPLM and affecting the generalization capability of the distilled model. This issue has not been adequately addressed in previous studies. Existing methods often overlook the importance of maintaining and utilizing the internal knowledge of the SPLM during the distillation process. This can result in problems such as inadequate protection of the model’s inherent knowledge or its inflexible application during inference. To address these challenges, we propose a novel CoT distillation method that incorporates LoRA Mixture of Experts (MoE) to dynamically adapt to inference tasks while fully leveraging the internal knowledge of the SPLM. Furthermore, we introduce a task-oriented balance constraint that optimizes both the out-of-distribution (OOD) generalization ability and in-domain (IND) reasoning performance of the SPLM.