Text recognition systems are essential for digitizing documents. Until now, handwritten recognition systems have been divided into two separate types: one that reads individual lines of text and the other that processes entire paragraphs. This division exists because of technical limitations, not because reading lines and paragraphs are fundamentally different tasks. While these distinct systems work well in their respective niches, this separation causes practical problems. For example, when a paragraph contains just one line of text, paragraph-reading systems typically cannot handle it properly. The management of two separate systems doubles the computer memory required, and organizations must procure two different systems that perform the same task. In this work, we present three key contributions. Firstly, our work demonstrates that the fundamental concepts of task arithmetic remain applicable despite not meeting all the prerequisites. We demonstrate that in the absence of a common pretrained model, synthetic ancestor models can be derived from existing models. Furthermore, we show that the application of task arithmetic principles yields a better synthetic ancestor model which can enhance performance across target tasks. Secondly, we present a unified model for paragraph and line-level recognition. This unified model manages to achieve a state-of-the-art paragraph Word Error Rate (WER) of 10.1% on the IAM dataset, while simultaneously maintaining a robust line WER of 11.3%. Lastly, we establish the first benchmark results where a unified model is tested on the combined paragraph and line IAM dataset.