This paper presents the design and development of a pretraining-based translation transformer model for Hindi, Bengali, and Maithili languages. Leveraging Byte Pair Encoding (BPE) tokenization, the model efficiently handles the unique morphological characteristics of these languages while significantly reducing vocabulary size. The training process is divided into three distinct phases to optimize memory usage and computational efficiency. Additionally, a weighted balanced loss function is employed to emphasize three critical aspects of translation: sparse categorical cross entropy, masked cross entropy, and a focused loss component targeting rare token prediction. This phased training strategy, coupled with the custom loss function, enables the model to efficiently learn from diverse linguistic data in resource-constrained environments, offering a robust solution for low-resource language translation. Finally, we get some good results in the form of faster convergence and an accuracy of 93% on NLLB dataset which is an increase of around 8–9% from state-of-art multilingual models.