The increasing volume of visual data in news archives and media sources poses significant challenges for efficient event retrieval. This paper presents a multimodal approach to tackle the problem of Event Retrieval from Visual Data. Our system integrates several techniques to process diverse query types, including text, image, and video. For image-text retrieval, the BLIP2 model is used to embed both images and text descriptions. In cases where queries are in Vietnamese, we employ the pre-trained VietAI/envit5-translation model to translate prompts into English before processing them with BLIP2. Object detection is handled by YOLOWorldv2, and text extraction from images utilizes PP-OCRv3 and VGG Transformer. Additionally, WhisperX is employed for audio-to-text conversion. Embeddings from textual data, whether derived from OCR or audio, are generated using sentence-transformers/all-MiniLM-L6-v2. These embeddings are indexed using Usearch, enabling fast and efficient retrieval. Furthermore, we developed a high-speed temporal search mechanism that calculates scores and combinations for consecutive related frames to improve performance in temporal queries. The system is capable of running efficiently on CPUs, with a maximum query processing time of 2 s for advanced queries, such as Temporal search, which require multiple models to run consecutively, making it a scalable solution for large-scale video data retrieval. Additionally, we have built a user-friendly interface using Streamlit, enabling users to easily interact with and utilize the system.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MAVERICS: Multimodal Advanced Visual Event Retrieval with Integrated CPU-Optimized Search

  • Tam Nguyen Vu Khai,
  • Bao Le Tran Gia,
  • Ngan Tran Kim Ngoc,
  • Khanh Hoang Ngoc,
  • Nguyen Pham Hoang Le,
  • Quyen Nguyen Huu,
  • Duy Phan The,
  • Van-Hau Pham

摘要

The increasing volume of visual data in news archives and media sources poses significant challenges for efficient event retrieval. This paper presents a multimodal approach to tackle the problem of Event Retrieval from Visual Data. Our system integrates several techniques to process diverse query types, including text, image, and video. For image-text retrieval, the BLIP2 model is used to embed both images and text descriptions. In cases where queries are in Vietnamese, we employ the pre-trained VietAI/envit5-translation model to translate prompts into English before processing them with BLIP2. Object detection is handled by YOLOWorldv2, and text extraction from images utilizes PP-OCRv3 and VGG Transformer. Additionally, WhisperX is employed for audio-to-text conversion. Embeddings from textual data, whether derived from OCR or audio, are generated using sentence-transformers/all-MiniLM-L6-v2. These embeddings are indexed using Usearch, enabling fast and efficient retrieval. Furthermore, we developed a high-speed temporal search mechanism that calculates scores and combinations for consecutive related frames to improve performance in temporal queries. The system is capable of running efficiently on CPUs, with a maximum query processing time of 2 s for advanced queries, such as Temporal search, which require multiple models to run consecutively, making it a scalable solution for large-scale video data retrieval. Additionally, we have built a user-friendly interface using Streamlit, enabling users to easily interact with and utilize the system.