Context-Aware Video Segment Search: Integrating Knowledge Graphs for Query-Based Retrieval
摘要
The exponential growth of video data has created significant challenges in retrieving relevant information from long-form multimedia content. Existing systems focus on short-form videos and rely on semantic vector search or shallow text matching, which struggle to capture contextual relationships in the video data. This study proposes a hybrid, context-aware retrieval framework that integrates neural text embeddings with knowledge graph based reasoning to improve accuracy and semantic relevance in video segment search based on a natural language user queries. The system performs contrastive segmentation and extracts entities and relations from video segments using the Relation Extraction By End-to-end Language generation (REBEL) model to store them in Neo4j. Retrieval combines graph traversal and vector similarity through a weighted fusion model, enabling both structural and semantic alignment with the user query. Experimental results demonstrate that hybrid retrieval significantly outperforms standalone semantic or graph-based methods, achieving the highest MRR of 0.84. Recall@K analysis further highlights the system’s ability to retrieve relevant content, with improvements observed as K increases. Overall, the framework offers an effective and scalable solution for multi-modal context-driven video search based on user query, with an opportunity for enhanced modality and LLM future extensions.