Radical Sequence Encoding with Fine-Tuned CLIP for Handwritten Chinese Character Recognition
摘要
Chinese characters are logographic in nature and possess complex compositional structures, making their recognition inherently challenging, particularly in handwritten form. Zero-shot recognition of Chinese characters further exacerbates this challenge. By leveraging the inherent structural properties of Chinese characters, it is possible to extract fine-grained structural information essential for effective recognition. Most current recognition methods rely on hard-coded representations of characters or radicals, overlooking the inherent flexibility and compositional similarity among them. This paper presents a novel radical encoding and information extraction module. The radical encoding module is built upon fine-tuning the CLIP model to learn continuous representations of radicals and their structural relationships. The information extraction module parses the radical sequence into a tree structure and extracts information to generate embeddings. We also evaluate the similarity between image features and embeddings. Our approach can also be used for zero-shot learning recognition tasks demonstrating superior performance compared to existing methods. To the best of our knowledge, this is the first work to fine-tune CLIP for continuous radical encoding while simultaneously integrating radical sequence information for enhanced recognition. Experimental results on zero-shot handwritten Chinese character recognition show that our method outperforms existing approaches, highlighting the effectiveness of the proposed methodology.