Scholarlink

The exponential growth of scientific literature necessitates advanced information retrieval systems leveraging diverse query modalities. This paper introduces a novel framework for multimodal retrieval of scientific articles, designed to process both textual and visual queries seamlessly. By utilizing Sentence Transformers for textual data and Vision Transformers (ViT) for analyzing scientific diagrams and images, the system embeds both modalities into a unified vector space within Elasticsearch. The proposed approach supports flexible retrieval tasks, such as retrieving relevant papers based on textual descriptions, visual inputs (e.g., charts or figures), or combined queries. Experiments were conducted, querying 150 queries on 111,694 scientific articles collected from the ‘unarXive’ data set. Comprehensive evaluations of scientific articles demonstrate the system’s efficacy, achieving a precision of 0.712, recall of 0.832, and F1-score of 0.767 for text-based queries. In contrast, visual queries yield a precision of 0.589 and recall of 0.751, surpassing baseline methods. For multimodal queries, the framework achieves a precision of 0.678 and a recall of 0.764, showcasing its robustness in handling heterogeneous inputs. The system holds significant potential for applications in academic search engines, enabling researchers to retrieve precise and contextually relevant scientific literature efficiently.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal Retrieval of Scientific Articles by Integrating Textual and Visual Embeddings

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Avaneesh Sundararajan,
  • Nikhil Karthik,
  • Nithish Chouti,
  • Udayini Vedantham,
  • Aayush Kumar,
  • Debaditya Kundu,
  • Krishnendu Ghosh

摘要

The exponential growth of scientific literature necessitates advanced information retrieval systems leveraging diverse query modalities. This paper introduces a novel framework for multimodal retrieval of scientific articles, designed to process both textual and visual queries seamlessly. By utilizing Sentence Transformers for textual data and Vision Transformers (ViT) for analyzing scientific diagrams and images, the system embeds both modalities into a unified vector space within Elasticsearch. The proposed approach supports flexible retrieval tasks, such as retrieving relevant papers based on textual descriptions, visual inputs (e.g., charts or figures), or combined queries. Experiments were conducted, querying 150 queries on 111,694 scientific articles collected from the ‘unarXive’ data set. Comprehensive evaluations of scientific articles demonstrate the system’s efficacy, achieving a precision of 0.712, recall of 0.832, and F1-score of 0.767 for text-based queries. In contrast, visual queries yield a precision of 0.589 and recall of 0.751, surpassing baseline methods. For multimodal queries, the framework achieves a precision of 0.678 and a recall of 0.764, showcasing its robustness in handling heterogeneous inputs. The system holds significant potential for applications in academic search engines, enabling researchers to retrieve precise and contextually relevant scientific literature efficiently.