Ancient Chinese brocade objects are often subjected to various kinds of deterioration, such as color fading, physical damage, and loss of patterns, which is challenging to handle in digital preservation. The paper presents a novel multi-scale deep learning architecture towards restoration of ancient brocade patterns, combining image enhancement and inpainting in one holistic framework. The introduced method employs multi-scale hierarchical feature extraction to well capture local details in patterns as well as global pattern configurations, and uses pattern-aware component to benefit from inherent patterns’ symmetries and repetitions. By utilizing the dual-task learning paradigm, the network simultaneously restores corrupted areas as well as recovers missing parts while retaining authenticity of the patterns. Comprehensive experiments on our collection of 5,000 ancient brocade images yield superior performance over state-of-the-art methods, with average improvements of 3.2 dB in terms of PSNR as well as 0.15 in SSIM. The comparisons with classical filtering methods, convolutional neural network-based algorithms, and transformer-based algorithms prove the efficiency of our multi-scale approach. Expert evaluations corroborate the ability of the framework to ensure historical authenticity while delivering aesthetically pleasing restorations. The paper lays groundwork between deep learning and digital humanities, giving a sound solution to cultural heritage preservation.