Ensuring the reliability of AI-based systems is a crucial challenge in today’s AI-driven environment. However, robustness is a key component of reliable AI, as the failure of these systems could have severe consequences in the critical domains such as in healthcare, transportation and finance. Eventually, the systems in these domains are largely dependent on various language models. Hence, measuring the robustness of those language models ultimately determines the end success though no such holistic framework is available for the same. Hence the current researchers are proposing a vulnerability detection framework which can be used to measure various vulnerabilities like jailbreak, prompt injection, biasness, data leakage and sensitive information disclosure through red teaming technique and ultimately publish a detailed report so that appropriate measures can be taken to address the same.