Generative Adversarial Networks (GAN) can produce realistic synthetic data that can be used in many applications including training other neural networks. However, a malicious GAN can intentionally exclude specific data while training the GAN model and it can create an incorrect neural network model. In this paper, we propose a secure GAN model where we partition the GAN model among two entities (a) a GAN provider (only who knows the discriminator model of GAN) and (b) a GAN user (who builds the generator model of the GAN). The GAN provider does not share its training data for the discriminator model but allows the GAN user to choose the type of data to be used to train the discriminator model. The GAN provider and the GAN user engage in oblivious transfer and zero-knowledge proof protocol to verify (a) the correct discriminator model is developed by the GAN provider, and (b) the correct discriminator model is used to train the generator model. We prove the proposed protocols for building GAN are secure and privacy-preserving. We also present an experimental evaluation of the proposed GAN.