HNSW Algorithm Explained: Hierarchical Navigable Small World for Similarity Search

M determines the number of connections per node in the HNSW graph (higher M = more connections = better precision but slower construction). EF_Construction controls candidate neighbors during index building (higher = more precise placement). EF_Search controls neighbors considered during query (higher = better precision but slower search). These parameters trade off between build time, query speed, and retrieval precision.

HNSW hyperparameters include M (number of edges between nodes) and EF construction/search (number of neighbors considered during graph traversal). Higher values improve recall but increase memory footprint. Binary quantization converts 32-bit floating-point vectors to single-bit representations, achieving significant compression at potential quality cost.

HNSW (Hierarchical Navigable Small World) is the dominant vector search algorithm due to its ease of use and good defaults. It uses two key parameters: M (bidirectional links per vector) and EF Search (candidate vectors during search). M controls graph density - higher M means more connections but slower construction. EF Search controls search radius - higher values scan more vectors but improve recall. HNSW builds layered graphs where top layers are sparse and bottom layers are dense. During insertion, vectors are placed by quickly traversing from top to bottom layers. During search, the algorithm starts at the top and converges to the bottom layer where it searches a larger radius. The default M=16 and EF Search=40 work well for most cases, with the embedding model having the biggest impact on search performance.

HNSW index construction uses two key parameters: (1) M - controls the number of links each vector maintains to other nearby vectors, determining neighborhood density; (2) EF Construction - controls the search radius during index building, affecting how thoroughly the graph is connected. Higher M values increase build time significantly but improve recall. Higher EF Construction values also improve recall but increase build time. Tuning these parameters involves balancing indexing cost against query performance and recall requirements.

HNSW indexes have one primary hyperparameter: the number of neighbors (M) maintained for each vector in the bottom layer. Increasing M improves precision but increases index size and search time. The relationship between M and performance is sublinear - doubling M roughly doubles index size and search time, but the precision improvement may not be proportional.

HNSW combines skip lists (probabilistic multi-layer linked lists) and navigable small world graphs (randomly inserted points with short-path connections). Search starts from the highest layer, finds nearest neighbors, and recursively descends to lower layers. Index selection depends on requirements: Flat for 100% recall, IVF for moderate scale, HNSW for large indexes. Milvus architecture is fully distributed with S3 storage, query nodes, data nodes, and index nodes, enabling scaling to billions of vectors through independent component scaling.

Milvus is a popular vector database suitable for production environments, supporting deployment on multiple machines for horizontal scaling with a Python client. It requires a two-step process: first insert data into a collection (with vectors and metadata), then create an index on that collection. Indexing can take significant time for large datasets. Qdrant is a vector database known for its user-friendly interface, supporting both in-memory operation (for local experiments) and persistent storage (for production). Qdrant collections can store multiple vector indexes with different configurations. HNSW (Hierarchical Navigable Small World) is a popular approximate nearest neighbor search algorithm that builds a graph where each point is connected to its k nearest neighbors. The graph is hierarchical, with points visible at different levels of the hierarchy. Each point is assigned a random level (from 0 to L), where L is a hyperparameter controlling the hierarchy depth. HNSW search starts at a random point at the highest level and navigates through the hierarchy, achieving O(log n) complexity. Vector quantization reduces storage requirements by compressing vectors, typically reducing size by 4-8x with minimal quality loss.

This section introduces FAISS (Fast Approximate Nearest Neighbor Search) as a vector search library. The presenter discusses pronunciation variations (fice, face, fast) and advocates for 'fast'. The core application demonstrated is building a question answering system where facts are stored in an HNSW vector database and questions serve as queries. The embedding model (mini LM) produces 384-dimensional vectors for both questions and facts. The HNSW implementation requires three key parameters: M (graph connectivity and number of layers), EF Construction (candidates considered during node insertion), and EF Search (candidates considered during retrieval). These parameters control the trade-off between index size, construction speed, and search accuracy.

The vector database ecosystem includes several major players: Milvus (self-hosted, supports multiple AN algorithms); Vespa (Yahoo, offers self-hosted and managed options with tensor data structures); Pinecone (fully managed, proprietary algorithms); Weaviate (GraphQL query syntax, stores inverted index, vector index, and original objects); Faiss (library implementing HNSW and other algorithms); GSI APU (hardware accelerator using associative processing units); Quadrant (open source, focuses on metric learning). Implementation languages vary from Go to Rust to Java to C++ to Python. Key differentiators include support for filters, live index updates, horizontal scaling, and integration with neural frameworks.

This section covers Retrieval Augmented Generation (RAG) and vector search fundamentals. RAG is a method to intelligently retrieve relevant data from a dataset to provide context to Large Language Models (LLMs), enabling personalized and accurate responses for specific use cases. Vector embeddings are compact semantically rich representations of data that enable semantic search, with similarity measured by distance functions like cosine similarity. Two vector indexes are supported: IVF (Inverted File Index) clusters data semantically and is fast to build and memory-efficient, ideal for small-scale projects; HNSW (Hierarchical Navigable Small World) uses a multi-layer graph for higher performance and accuracy on large datasets (trillions of data points) with better handling of insertions/deletions, though it requires more memory.

Retrieval-Augmented Generation (RAG) is an architecture that addresses LLM limitations by using vector search to retrieve relevant documents based on input queries, then providing these retrieved documents as context to the language model to help generate more informed and accurate responses. Instead of generating responses purely from patterns learned during training, RAG uses retrieved documents to ground the model's responses in factual information. This minimizes hallucinations, ensures responses reflect current information, allows utilization of external databases, and makes token usage more efficient by retrieving only relevant documents.

This section demonstrates the complete RAG pipeline from user query to final LLM response. The instructor explains the vector search process: encoding user queries into embeddings using the same model used for document embeddings, using the vector index to find similar documents, and retrieving results with similarity scores. The section covers the MongoDB aggregate method with vector search pipeline configuration, including vector search settings, vector index information, and query embedding. The instructor demonstrates retrieving results with similarity scores and explains how results include both retrieved documents and their similarity scores. The section also covers troubleshooting common issues like incorrect column names, vector index not activated, and collection not properly selected. The final part demonstrates combining retrieved documents (full plot, title, year, score) with the original user query to create a complete prompt for the LLM. The instructor covers loading LLM models from Hugging Face (using Gemma 2B), obtaining Hugging Face tokens, configuring the model, tokenizing the combined prompt, and generating responses. The final output shows how the RAG system provides context-aware answers based on retrieved knowledge, demonstrating the complete end-to-end workflow.

RAG represents the most popular retrieval-augmented generation approach with a three-component pipeline: (1) Indexer - indexes all documents into a vector database; (2) Retriever - when given a query, retrieves top-K relevant passages from the index; (3) Generator - concatenates retrieved passages with the query and generates the final response. The key advantage is flexibility - any off-the-shelf retriever (including Google search) and any pre-trained language model can be used. Popular implementation tools include LangChain and LlamaIndex. This approach enables practical deployment of retrieval-augmented systems in industry settings.

The complete RAG pipeline demonstrates retrieving context from vectorized documents. When a user query is passed to the retriever, it generates a query embedding, queries the vector store, and returns the most relevant documents with their metadata and similarity scores. This enables dynamic context retrieval where the system fetches the most pertinent information from the stored knowledge base to augment LLM responses, forming the basis for powerful retrieval-augmented generation systems.

The core of vector database performance lies in indexing algorithms that power approximate nearest neighbor search. HNSW (Hierarchical Navigable Small World) creates multi-layered graph structures with connections between vectors at different levels, offering excellent query speed with 95%+ recall accuracy but requiring significant memory (50-100 bytes per vector) and query times of 1-5ms for million-scale datasets. IVF (Inverted File Index) partitions vector space into clusters and searches only the most relevant clusters, scaling better to billion-scale datasets but trading some accuracy; the n_list parameter (number of clusters) should be set to the square root of the dataset size. Product Quantization compresses vectors by encoding them into smaller representations, reducing memory by 75-95% but introducing quantization error. Tree-based approaches like ANNOY build forest-like structures suitable for disk-based deployments. Algorithm selection should be data-dependent: for 10+ million vectors, consider IVF-PQ; for less than 1 million vectors where accuracy is critical, HNSW typically delivers best results.

ANN can be implemented through multiple strategies: tree-based approaches like ANNOY (used by Spotify), graph-based approaches like HNSW (Hierarchical Navigable Small World), cluster-based approaches like FAISS (Facebook AI Similarity Search), product quantization (PQ), and hash-based approaches like LSH (Locality Sensitive Hashing). HNSW creates nodes representing data points with edges connecting to similar vectors, building a hierarchical structure where clusters become nodes at higher levels. This enables efficient traversal during search by hopping through clusters and nodes to find similar items quickly, making it one of the most popular ANN indexing strategies.

ANNOY (Approximate Nearest Neighbors Oh Yeah) partitions vectors by randomly selecting two vectors and assigning all others to the closer one, recursively creating tree structures. Multiple trees improve recall by covering different partitions. IVF (Inverted File) uses K-means clustering to partition vectors into K groups with centroids, enabling horizontal scaling across nodes. Each node handles one cluster, routing queries to appropriate nodes. This partitioning reduces search space dramatically, though IVF alone doesn't address the curse of dimensionality that makes tree-based approaches ineffective in high dimensions.

IVF Flat (Inverted File Flat) builds clusters by sampling ~10,000 vectors using k-means, then assigning all vectors to nearest centers. Querying involves visiting one or more cluster centers and comparing against all vectors in those clusters. More centers visited improves recall but increases I/O. HNSW (Hierarchical Navigable Small Worlds) builds a multi-layered graph where top layers are sparse and bottom layers are dense. Each node maintains bidirectional links to m neighbors (default 16). Search starts at top layers, progressively moving down while maintaining visited lists and candidate queues. Higher m improves recall but increases query time and index size. Both methods require careful parameter tuning to balance search quality against performance.

Milvus supports three index categories: quantization-based (IVF_FLAT, IVF_SQ8, IVF_SQ8H, IVF_PQ), graph-based (RNSG, HNSW), and tree-based (ANNOY). IVF indexes use NLIST for cell count and M for PQ decomposition. RNSG uses knn, search length, candidate pool size, and out degree. HNSW uses M and efConstruction. ANNOY uses entries for tree count. GPU index building is available via gpu.enabled parameter. Index building is computationally intensive, requiring careful parameter balancing between build speed and query recall.