<p>Approximate nearest neighbor search (ANNS) is a widely used technique in data retrieval, yet its performance is often compromised by the presence of outliers, which can significantly distort the indexing structure and retrieval results. To address this issue, we propose <Emphasis FontCategory="NonProportional">BraveANN</Emphasis>, a memory-disk hybrid vector indexing and search system designed to improve robustness against outliers while maintaining low latency. The core of <Emphasis FontCategory="NonProportional">BraveANN</Emphasis> is our newly proposed clustering approach, <Emphasis FontCategory="NonProportional">RKM</Emphasis>, which utilizes exponential tilting to perform robust <i>k</i>-means clustering. While <Emphasis FontCategory="NonProportional">RKM</Emphasis> improves outlier robustness by positioning centroids further away from outliers, it introduces challenges of speed and imbalance in index construction. To mitigate these issues, we propose an accelerated algorithm, <Emphasis FontCategory="NonProportional">FastRKM</Emphasis>, and a balanced algorithm, <Emphasis FontCategory="NonProportional">BalanceRKM</Emphasis>, which enhance <Emphasis FontCategory="NonProportional">BraveANN</Emphasis> by speeding up index construction and reducing latency. Our experimental results consistently demonstrate that our method outperforms SOTA methods in both effectiveness and efficiency across eight real-world datasets and six metrics. Specifically, on billion-scale datasets, <Emphasis FontCategory="NonProportional">BraveANN</Emphasis> achieves up to a 3.14-5.48% improvement in recall compared to other memory-disk hybrid methods. Moreover, as the proportion of outliers increases, <Emphasis FontCategory="NonProportional">BraveANN</Emphasis> exhibits a slower decline in recall, highlighting its robustness. Compared to memory-based methods, <Emphasis FontCategory="NonProportional">BraveANN</Emphasis> significantly reduces memory usage while ensuring high recall and low latency.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

BraveANN: Robust Approximate Nearest Neighbor Search for Billion-Scale Vectors

  • Shengkun Zhu,
  • Yiming Wang,
  • Xin Jin,
  • Jinshan Zeng,
  • Sheng Wang,
  • Yuan Sun,
  • Yuhui Lai,
  • Zhiyong Peng

摘要

Approximate nearest neighbor search (ANNS) is a widely used technique in data retrieval, yet its performance is often compromised by the presence of outliers, which can significantly distort the indexing structure and retrieval results. To address this issue, we propose BraveANN, a memory-disk hybrid vector indexing and search system designed to improve robustness against outliers while maintaining low latency. The core of BraveANN is our newly proposed clustering approach, RKM, which utilizes exponential tilting to perform robust k-means clustering. While RKM improves outlier robustness by positioning centroids further away from outliers, it introduces challenges of speed and imbalance in index construction. To mitigate these issues, we propose an accelerated algorithm, FastRKM, and a balanced algorithm, BalanceRKM, which enhance BraveANN by speeding up index construction and reducing latency. Our experimental results consistently demonstrate that our method outperforms SOTA methods in both effectiveness and efficiency across eight real-world datasets and six metrics. Specifically, on billion-scale datasets, BraveANN achieves up to a 3.14-5.48% improvement in recall compared to other memory-disk hybrid methods. Moreover, as the proportion of outliers increases, BraveANN exhibits a slower decline in recall, highlighting its robustness. Compared to memory-based methods, BraveANN significantly reduces memory usage while ensuring high recall and low latency.