Generating handwritten Chinese characters is a critical yet challenging task in computer vision due to their structural complexity and vast quantity. Constructing a comprehensive Chinese character library is both time-consuming and labor-intensive. While existing models mitigate the demand for large-scale character datasets, they often struggle with semantic accuracy and generation efficiency. In this paper, we propose HandDiff-GAN, a novel model that combines Generative Adversarial Networks (GANs) with a pre-trained conditional denoising diffusion probabilistic model (DDPM), aiming to generate high-quality handwritten Chinese characters using only a small number of reference samples. Furthermore, the model incorporates an interactive style adjustment mechanism, enabling users to fine-tune character styles in real time by adjusting features such as stroke thickness, tilt, and pen pressure. This user-driven approach ensures the generated characters align closely with user preferences. The proposed model preserves the style features of the reference samples while maintaining the structural integrity of the generated characters. This user-driven adjustment mechanism adds great flexibility to the generation process and enhances the practicality and robustness of the model. HandDiff-GAN has demonstrated great potential for personalized handwritten font generation, providing a flexible, efficient, and friendly solution for character generation.