High-dimensional similarity search plays a critical role in large-scale information retrieval systems, yet traditional brute-force search methods are computationally expensive and impractical for real-time applications. Approximate search techniques such as hashing and clustering improve efficiency but often sacrifice accuracy. To address these challenges, we propose SHARP (Speed-Enhanced High Accuracy Retrieval Process), a novel two-stage cosine similarity search technique that balances efficiency and precision in vector databases. Our approach first applies Principal Component Analysis (PCA) to reduce the dimensionality of sentence embeddings, enabling a fast initial search in a lower-dimensional space. The most relevant candidates from this stage are then refined using a full-resolution cosine similarity search in the original high-dimensional space, ensuring high retrieval accuracy. This method significantly reduces computational overhead while preserving search quality. Experimental evaluations on large-scale document datasets demonstrate that SHARP achieves a 70% reduction in search time compared to single-stage methods while maintaining a high retrieval accuracy of 98.1%. Furthermore, our approach scales efficiently across increasing dataset sizes, making it well-suited for real-time document retrieval, search engines, and recommendation systems. The results highlight that SHARP provides an effective trade-off between computational speed and retrieval performance, offering a scalable and accurate solution for modern vector search applications such as RAG systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SHARP: Speed-Enhanced High Accuracy Retrieval Process Using Two-Stage Cosine Search Similarity

  • Pandey Shourya Prasad,
  • Ritik Kumar Gupta,
  • B. Thangaraju

摘要

High-dimensional similarity search plays a critical role in large-scale information retrieval systems, yet traditional brute-force search methods are computationally expensive and impractical for real-time applications. Approximate search techniques such as hashing and clustering improve efficiency but often sacrifice accuracy. To address these challenges, we propose SHARP (Speed-Enhanced High Accuracy Retrieval Process), a novel two-stage cosine similarity search technique that balances efficiency and precision in vector databases. Our approach first applies Principal Component Analysis (PCA) to reduce the dimensionality of sentence embeddings, enabling a fast initial search in a lower-dimensional space. The most relevant candidates from this stage are then refined using a full-resolution cosine similarity search in the original high-dimensional space, ensuring high retrieval accuracy. This method significantly reduces computational overhead while preserving search quality. Experimental evaluations on large-scale document datasets demonstrate that SHARP achieves a 70% reduction in search time compared to single-stage methods while maintaining a high retrieval accuracy of 98.1%. Furthermore, our approach scales efficiently across increasing dataset sizes, making it well-suited for real-time document retrieval, search engines, and recommendation systems. The results highlight that SHARP provides an effective trade-off between computational speed and retrieval performance, offering a scalable and accurate solution for modern vector search applications such as RAG systems.