The well-trained Generative Adversarial Networks (GANs) are capable of generating high-quality images. However, training a GAN with limited data poses a significant challenge. Notably, in scenarios with only one reference, the existing methods of fine-tuning a pre-trained GAN are prone to mode collapse. To address this issue, we continuously expand the dataset by combining GAN inversion, the style mixing capability of StyleGAN, and manually specified masks. We introduce the above method into two-stage training, including fine-tuning discriminator and one-shot domain adaptation. During GAN inversion, we design a latent code mapper to map the latent code to the target domain, which improves the editability of results compared with the traditional optimization-based methods. To enhance details of results, we propose an adaptive feature selection loss that dynamically selects features and constrains them to the source or target domain. Qualitative and quantitative experiments show that our method is superior to the state-of-the-art methods in one-shot tasks, particularly in terms of diversity.