This paper introduces a digital restoration method specifically developed for ancient painted Caihua, the traditional ornamentation commonly found in Chinese architectural heritage. Caihua are significant cultural artifacts that often deteriorate due to aging, environmental factors, and human-induced damage. To avoid risks of physical intervention, we propose a deep learning-based framework that automatically detects, segments, and digitally restores defects in caihua patterns in a non-destructive manner. Our approach integrates convolutional neural networks (CNNs) with Transformer architectures, combining local feature extraction with global contextual understanding. A hybrid CNN–Transformer model analyzes complex patterns and enables precise segmentation of damaged regions. To improve edge preservation and color consistency, we introduce a light-weighted histogram shifting embedding, and employ graph pooling with attention to capture intricate textures and global structural context. Compared with PDE-based and single-paradigm deep models, the integration improves visual authenticity and artistic fidelity while remaining computationally practical. Extensive experiments and comparisons with state-of-the-art techniques (including a pure Transformer baseline) show superior restoration in visual detail, defect detection, and color fidelity. The automated, non-invasive pipeline reduces manual effort and supports cultural heritage preservation.