Various degradations available in historical document images pose challenges to automatic document restoration methods. Recent strides in deep learning and generative adversarial networks (GANs) for document analysis encounter challenges in generalization and practical application. Due to the absence of boundary-related supervision, conventional convolutional approaches often struggle to extract stroke edges from documents undergoing binarization consistently. In this study, we present a GAN-based robust end-to-end system to restore heavily damaged document images and tackle the problem of inaccurate stroke edge extraction. In the proposed framework, there is a single generator along with two discriminators. This approach leverages object and contour data concurrently, enhancing binarization results. The auxiliary discriminator is crucial in ensuring the accurate prediction of stroke edges. The solution consistently outperforms state-of-the-art techniques on DIBCO datasets from 2017 and 2018, proving its ability to recover a damaged document image. The results obtained across a diverse set of degradations showcase the flexibility of the proposed approach, making it applicable to various document restoration applications.