Ontology-Enhanced Semantic Image Search with Deep Learning and CLIP Embedding
摘要
This paper proposes a novel hybrid framework that enhances semantic image retrieval by integrating deep learning models with ontology-based reasoning. The system combines YOLOv8 for object detection, CLIP for generating joint visual–textual embeddings, and a domain-specific ontology automatically constructed from COCO 2017 and Visual Genome 2016 datasets. Semantic queries are executed using SPARQL over the ontology to enable explainable, logic-based filtering, while FAISS with HNSW indexing ensures scalable and efficient embedding search. We further leverage NLP models (BERT, T5) and query augmentation (NLPaug) to improve natural language understanding and query reformulation. Experimental results on a benchmark of 30,000 images and 500 diverse user queries show that our approach consistently outperforms baseline and state-of-the-art methods in terms of Precision@10, Recall@10, mAP, and F1-score. Notably, our system achieves a strong balance between accuracy and response time, demonstrating the effectiveness of combining symbolic knowledge with deep embeddings for interpretable, high-performance image retrieval.