The widespread use of Large Language models (LLMs) raises concerns about their robustness and safety in critical systems. Software testing reveals vulnerabilities that can be addressed to enhance the quality of software products. However, due to the non-deterministic nature of LLMs’ architecture, the adoption of traditional testing methods in this domain is not straightforward. Consequently, novel testing techniques are being adopted to reveal vulnerabilities of LLMs. However, most of these techniques know little about the internal architecture of the LLM being tested. In this context, Model-Based Testing (MBT) can represent aspects of the LLM’s internal structure to guide Test Case Generation (TCG). This paper proposes a conceptual framework for applying MBT to improve testing techniques and ensure the robustness and safety of LLMs. This framework adapts different testing approaches from the machine learning domain. Thus, testing could be performed systematically at a conceptual level based on LLM architecture models by applying techniques such as neuron coverage and mutation testing.