There are some ancient documents that reuse paper due to the rarity of paper in the past, and that have different documents written on both sides. Among these documents, the ones that were first no longer needed and were left on the back side of the paper are called “Shihai-monjo,” and it is known that historically important content has been left in them as well. However, it is difficult to read the Shihai-monjo by peering into the interior of a document with a bag-bound structure, from the perspective of preserving the document. In this study, we developed an image restoration system based on a deep learning-based image inpainting method to solve the problem of missing back side characters caused by overlapping characters, which is an issue when analyzing ancient document images taken from the top.