Text simplification involves reducing complexity while retaining important information. This is important for improving accessibility for a wide range of readers, including those with cognitive disorders, non-native speakers, and children. We report experiments on text simplification for Lithuanian with a focus on simplifying texts, written in an administrative style, which is not easy to understand for the general public. We chose mT5 as a foundational model and fine-tuned it in 3 different ways. We evaluated models’ outputs with ’unseen’ data and automatic evaluation metrics accompanied by qualitative analysis. Though automatic evaluation scores were higher for the model fine-tuned with translated data, qualitative analysis revealed that the model that was fine-tuned with specifically for this task prepared Lithuanian data performed simplification task significantly better.