Object detection is a critical task in computer vision that can be hampered by limited training data. To overcome data insufficiency, data augmentation has been widely used, traditionally applying geometric transformations or pixel manipulations to existing images. However, these methods lack diversity as they do not introduce new information fundamentally. Text-to-image algorithms, which combine diffusion models with prompts, can generate a more diverse range of images. However, due to potential inaccuracies or redundancies in the prompts, the generated images may deviate from reality. Simultaneously, constructing effective prompts remains a challenge, as it requires carefully selecting relevant keywords while eliminating redundant information. To tackle these limitations, we introduce a novel method called DiffGAN, which merges the diversity capabilities of diffusion models with the reality-enhancing mechanisms of Generative Adversarial Networks (GANs) in an adversarial learning process. DiffGAN utilizes a GAN generator to create text features aligned with the desired image and a GAN discriminator to assess the reality of the images produced by the diffusion model, thereby iteratively refining the GAN generator. Our experiments on the Open Images Dataset V3 and a Diseased Leaf dataset demonstrate that DiffGAN outperforms traditional data augmentation methods, improving object detection metrics (mAP@0.5) by 13.9% and 16.1%, respectively.