Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs

, , , , The paper introduces a new algorithm called Hierarchical Navigable Small World (HNSW) for the approximate nearest neighbor search. HNSW is a fully graph-based approach that utilizes navigable small world graphs, allowing for simple insertion, deletion, and K-nearest neighbor queries. Unlike other proximity graph techniques that rely on additional search structures like kd-trees or Cartesian concatenation, HNSW constructs a layered structure consisting of hierarchical sets of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is randomly selected using an exponentially decaying probability distribution. This ensures that the resulting graphs are similar to Navigable Small World (NSW) structures while also having links separated by their characteristic distance scales. By starting the search from the upper layer instead of random seeds and utilizing scale separation, HNSW achieves better performance compared to NSW and allows for logarithmic complexity scaling. Additionally, a simple heuristic is employed to select proximity graph neighbors, further improving performance at high recall and in cases of highly clustered data. The proposed algorithm outperforms previous state-of-the-art vector-only approaches such as FLANN, FALCONN, and Annoy in terms of performance evaluation on various datasets. Its similarity to a well-known 1D skip list structure enables straightforward efficient and balanced distributed implementation. In summary, HNSW presents an efficient and robust solution for approximate nearest neighbor search in metric spaces. Its fully graph-based approach without reliance on additional search structures makes it versatile and applicable to various domains.

Introduction

The search for nearest neighbors is a fundamental problem in many fields, including data mining, machine learning, and information retrieval. It involves finding the closest points to a given query point in a high-dimensional space. This task becomes increasingly challenging as the dimensionality of the data increases. Traditional methods such as linear search become inefficient due to their computational complexity scaling with the size of the dataset.

To address this issue, researchers have proposed various approximate nearest neighbor (ANN) algorithms that aim to find an approximate solution with lower computational cost. One such algorithm is Hierarchical Navigable Small World (HNSW), which was introduced in a research paper titled "Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs" by Yury Malkov and Dmytro Ponomarev.

In this blog article, we will delve into the details of HNSW and its contributions to ANN search.

The Need for Approximate Nearest Neighbor Search

As mentioned earlier, traditional methods like linear search are not efficient when dealing with high-dimensional datasets. In these cases, it becomes necessary to use approximation techniques that can provide results close enough to the exact solution while significantly reducing computation time.

Approximate nearest neighbor search has applications in various domains such as image recognition, natural language processing, recommendation systems, and more. For example, in image recognition tasks where millions of images need to be compared against each other for similarity or clustering purposes, ANN algorithms can greatly improve efficiency without compromising accuracy.

The HNSW Algorithm

HNSW is a fully graph-based approach that utilizes navigable small world graphs for approximate nearest neighbor search. The key idea behind HNSW is constructing hierarchical sets of proximity graphs (layers) for nested subsets of stored elements instead of relying on additional structures like kd-trees or Cartesian concatenation.

The algorithm starts by randomly selecting a maximum layer for each element using an exponentially decaying probability distribution. This ensures that the resulting graphs are similar to Navigable Small World (NSW) structures while also having links separated by their characteristic distance scales. The elements are then inserted into the corresponding layers based on their maximum layer, with higher layers containing fewer elements and thus providing better scale separation.

Navigable Small World Graphs

Navigable small world graphs are a type of proximity graph that allows for simple insertion, deletion, and K-nearest neighbor queries. These graphs have two key properties: navigability and small-worldness.

Navigability refers to the ability to quickly reach any point in the graph from any other point through short paths. In HNSW, this is achieved by starting the search from the upper layer instead of random seeds.

Small-worldness refers to the property where most nodes in a graph can be reached within a few hops from any other node. This is ensured in HNSW by constructing hierarchical sets of proximity graphs with decreasing size as we move up the layers.

Improvements over NSW

HNSW builds upon NSW's structure but introduces some improvements that make it more efficient and robust. Firstly, HNSW employs a simple heuristic to select proximity graph neighbors based on their distance rather than randomly choosing them like in NSW. This improves performance at high recall and for highly clustered data.

Secondly, HNSW utilizes scale separation between layers, which allows for logarithmic complexity scaling compared to linear scaling in NSW. This makes HNSW more suitable for large datasets as its performance does not degrade significantly with increasing dataset size.

Evaluation Results

To evaluate its performance, HNSW was compared against previous state-of-the-art vector-only approaches such as FLANN, FALCONN, and Annoy on various datasets. The results showed that HNSW outperformed these algorithms in terms of efficiency and accuracy.

Additionally, the similarity between HNSW and a well-known 1D skip list structure enables straightforward efficient and balanced distributed implementation. This makes it versatile and applicable to various domains.

Conclusion

In conclusion, HNSW presents an efficient and robust solution for approximate nearest neighbor search in metric spaces. Its fully graph-based approach without reliance on additional search structures makes it versatile and applicable to various domains. With its improvements over NSW, HNSW outperforms previous state-of-the-art vector-only approaches in terms of performance evaluation on various datasets. Its logarithmic complexity scaling also makes it suitable for large datasets, making it a valuable addition to the field of ANN search algorithms.

If you are interested in learning more about this algorithm, we highly recommend reading the original research paper by Malkov and Ponomarev. We hope this blog article has provided you with a better understanding of HNSW and its contributions to approximate nearest neighbor search.