Model collapse in large language models (LLMs) and mode collapse in generative adversarial networks (GANs) refer to the phenomena where these AI models fail to generate diverse and high-quality outputs, which in turn undermines their effectiveness. Collapse can mainly occur due to a large bias or a lack of diversity in training data, which is particularly relevant in future use cases of such models due to the huge amounts of AI-generated, or self-generated, data present in today’s training sets. As a result, the reliability and scalability of generative AI models in the future could be significantly affected, which could in turn limit their ability to generalize outputs and perform across a wide range of tasks. This paper offers a thorough analysis of these issues by using an in-depth technical analysis of various GAN and LLM architectures to identify the main factors that lead to collapse which include optimization problems, data biases and imbalance, problems within the architecture and flawed training algorithms. This paper also provides an extensive review of existing works in this domain to identify any potential approaches that could be used to solve this issue, and hence, also aims to guide any future advancements in generative AI and enhance the reliability and efficiency of such generative models.