Paraphrases are texts written using different words but conveying the same meaning; hence, their quality is based upon retaining semantics while varying syntax/vocabulary. Recent works leverage structured syntactic information to control the syntax of the generations while relying on pretrained language models to retain the semantics. However, rarely do works in the literature consider using structured semantic information to enrich the language representation. In this work, we propose to model the task of paraphrase generation as a pseudo-Graph-to-Text task where we fine-tune pretrained language models using as input linearized representations of pseudo-semantic graphs built from dependency parsing trees sourced from the original input texts. Our model achieves competitive results on three popular paraphrase generation benchmarks.