Molecular design is one of the challenging problems in chemical synthesis due to the large search space of valid molecules. Existing methods are based on two key encoding approaches: molecular graph and textual SMILES. Molecular graph encoding methods are expressive and chemically-aware as they include atoms, bonds and other molecular properties. SMILES-based approaches on the other hand do not consider any chemical information and treat the molecules as a sequence of characters. Current generative molecular graphs and SMILES-based models learn the distribution of the input and then sample from the learned distribution to generate new molecules. SMILES-based methods are prone to generating invalid molecules and are not chemically aware. Despite this however, the success of Large Language Models (LLMs) in Natural Language Processing (NLP) has led to the development of strong LLM methods which are competitive with the state-of-the-art molecular graph-based methods. This paper shows how a fragment-based SMILES LLM can be trained and sampled effectively with beam search to improve the generated molecules’ validity, novelty and uniqueness. We evaluate the model on two standard molecular design datasets: ZINC and PCBA. We show that our model can generate accurate molecules with high validity, novelty and uniqueness while recording results comparable to or better than the state-of-the-art molecular graph-based methods.