Multimodal Tattoo Recognition by Combining Visual Features and LLM-Generated Captions
摘要
Tattoo recognition and retrieval remain challenging tasks due to the diverse and intricate nature of tattoo designs. Existing approaches typically rely on visual features extracted from convolutional neural networks (CNNs), which may fail to capture the rich semantic information embedded in tattoos. In this paper, we propose a novel framework that integrates state-of-the-art visual features with textual descriptions generated by a multimodal large language model (MLLM). We explore the impact of different prompts on the quality of the generated captions and use CLIP to create textual embeddings. By combining cosine similarity scores from both modalities, our approach achieves superior performance in tattoo retrieval tasks. Experimental results demonstrate that our method outperforms traditional visual-only approaches, highlighting the importance of leveraging multimodal data for tattoo recognition. To the best of our knowledge, this is the first work to combine MLLM-generated textual descriptions with visual features for tattoo retrieval.