Handwritten-Text-Image Editing with Diffusion Models and Attention-Based Matching
摘要
Advancements in diffusion models have significantly enhanced image-generation, with emerging applications extending into multimodal domains. However, persistent challenges in handwritten-text generation—particularly in maintaining typographic consistency and enabling precise modifications—remain unaddressed. This paper presents a novel diffusion model that leverages handwritten-glyph embeddings and attention-guided localization to achieve high-fidelity handwritten-text generation and editing. The proposed model consists of three key modules: (1) a handwritten-glyph-embedding module that incorporates style features and content features for text-control diffusion loss, (2) low-rank decomposition module that enables style-consistent modifications through optimized matrix injections, and (3) attention-guided localization module integrated with segmentation-based text-region detection for editable region detection. Experimental validation demonstrates substantial improvements over other existing methods: our attention-guided localization module achieved 20% higher mean reciprocal rank and 60% lower miss detection rate compared with conventional optical character recognition (OCR) models. In generation-quality assessment, the proposed model outperformed state-of-the-art baselines such as AnyText, showing improvement in learned perceptual image patch similarity and scale-invariant feature transform metrics. The proposed model enables interactive local editing while preserving non-target regions, establishing new benchmarks for style-consistent handwritten-text manipulation.