Code Large Language Models (LLMs) have demonstrated considerable capabilities in software engineering. One major approach to utilizing code LLMs is through Supervised Fine-Tuning (SFT), which enhances the model's capabilities for specific tasks, such as improving its adherence to user instructions for code generation. However, it has been observed that SFT can cause the degradation of certain fundamental capabilities of general LLMs due to catastrophic forgetting. In this paper, we propose Bayesian Auto Merging (BAM), a novel method that leverages model merging and Bayesian optimization to mitigate this degradation by enhancing the performance of code LLMs post-SFT. We first demonstrate that SFT can lead to a decline in the coding and instruction-following capabilities of code LLMs. To address this, BAM merges pre-SFT and post-SFT models, using the pre-SFT model to restore the capabilities of the post-SFT model. Model merging, however, heavily depends on the merging parameters, such as the weight parameter that defines the mix ratio between models. Identifying the optimal configuration for these parameters is both labor-intensive and complex. To overcome this challenge, BAM utilizes Bayesian optimization to automatically search for the optimal configuration of these parameters. Empirical evaluations on the HumanEval and MBPP datasets with Qwen2.5-Coder and DeepSeek-Coder models show that BAM consistently improves performance(e.g., enhancing the pass@1 metric by an average of 1.6% on HumanEval). All the experimental code and instructions for reproducing the essential results can be found at https://github.com/Chengzhe-Feng/BAM .