<p>Pre-trained and Large Language Models (LLMs) such as GPT and BERT can learn new tasks after being fine-tuned on a new set of dataset. They also can learn multimodal features and exhibit outstanding results on tasks such as image captioning and visual question-answering. These models represent exceptional performance once are fine-tuned with data collected and engineered by humans. However, the process of data collection and annotation is an expensive process in terms of time and resources. In addition, using real data may raise additional concerns in terms of privacy and security. There are some techniques such as data cleaning and data augmentation to increase the quality and quantity of data. However, these techniques cannot produce reliable and diverse datasets. Diffusion models have recently received attention for generating synthetic data. In this paper, we investigate the capacity of various diffusion models and LLMs to generate a broad spectrum of data with high fidelity and diversity for vision-language models. We evaluate the performance of vision-language models on synthetic data for zero-shot learning, few-shot learning, and pre-training. Our finding shows the competence of diffusion models to generate synthetic data for vision-language models on tasks such as image captioning and visual question-answering. We empirically show that this ability is integrated with the capacity of large language models to create a diverse range of reliable captions for diffusion models and vision-language models.</p>