Large text-to-image models have demonstrated remarkable performance in the domain of image synthesis, enabling the generation of high-resolution and diverse images of superior quality from an input text prompt. However, achieving such performance of image synthesis heavily relies on substantial computation resources and extensive datasets, which may not be accessible to individuals. What’s more, leveraging large pre-trained models effectively is challenging if the target domain is significantly small compared to their original domain. In this work, we propose a hybrid pipeline that leverages both large text-to-image diffusion models aimed at domain transferring and lightweight networks to provide extra guidance for our approach. Given a few target domain images and a large pre-train diffusion model, we can fine-tune the model in a partially frozen manner and improve the image quality of the given domain. The extra guidance used is a canny edge method since we believe the visual guidance is more informative than embedded special tokens. In comparison, our model generated better and more realistic images. Implementation of our proposed model is publicly available at https://github.com/ZakirANU/Generating-Realistic-Images .