Few-shot cross-modal text detection via CLIP
摘要
Text detection presents a significant challenge in the field of computer vision, particularly due to the arbitrary shapes and domain gaps of text instances. Most existing methods for developing accurate text detectors depend on large amounts of finely annotated data. This dependence not only leads to significant annotation expenses but also limits the generalization of these methods to broader scenarios. The cross-modal approach, exemplified by Contrastive Language-Image Pre-training (CLIP), effectively enhances the model’s few-shot learning and generalization capabilities by aligning textual and visual modality information. However, aligning textual and visual information for the entire image using CLIP may result in the model’s inability to understand text instances in a fine-grained manner. In addition, the limited text input length of CLIP is insufficient for describing the variable scene text instances. Therefore, this paper proposes a novel few-shot cross-modal text detection method based on CLIP, incorporating three key improvements: (1) a parallel-stem image encoder, (2) a lower-position frozen position embedding interpolation method, and (3) a cross-modal feature propagation mechanism. Evaluation results demonstrate that our method achieves the best few-shot performance. For example, with 20% labeled data, the F-measure exceeds the baseline by 11.14 on the ICDAR 2015 dataset. Additionally, the method shows excellent generalization across several benchmarks.