Handwritten document image binarization is a semantic segmentation process that differentiates ink pixels from background pixels of the surface material. It is an essential step toward several handwriting analysis tasks, including character recognition, writer identification, and estimating the development of script styles. The binarization task is challenging due to the vast diversity of writing styles among people, writing surfaces, and inks used. It is even more complex for ancient historical manuscripts due to the aging and degradation of these documents over time. One such manuscript collection is the famous Dead Sea Scrolls (DSS), which poses extreme challenges for the existing binarization techniques. This article presents a novel binarization pipeline for DSS images using deep encoder-decoder networks. Although the artificial neural network presented here is primarily designed to binarize the DSS images, it can also be used on many other manuscript collections. Additionally, transfer learning demonstrates the network’s versatility for a wide range of handwritten documents, making it a unique, multi-purpose tool for binarization. Qualitative results and several quantitative comparisons using historical manuscripts and datasets from handwritten document image binarization competitions (H-DIBCO and DIBCO) exhibit the system’s robustness and effectiveness. The best-performing network architecture proposed here, BiNet, derived from the U-Net encoder-decoder, outperforms existing methods by achieving an F-score of 86.7% (±9.4) on fused images created using multi-spectral photos of the DSS collection.