Efficient Serving with ScaNN for Approximate Nearest Neighbor Search

The choice of vector database depends on product lifecycle stage, budget, scale, and DevOps capabilities. For early-stage/PoC with <1 million vectors, Chroma DB or Qdrant are recommended due to their low cost ($0-$50/month), ease of setup, and ability to migrate later. For production/midscale with 1-10 million vectors, Pinecone (managed, $70-$300/month) or Weaviate (self-hosted, $70-$5,500/month) provide production-ready infrastructure with multi-tenancy support. For enterprise/large scale (>10 million vectors), Weaviate (self-hosted) or Milvus (distributed, open-source) are recommended for distributed architecture and fine-grained control. At 100+ million vectors, managed solutions exceed $10,000/month.

Pinecone is a fully managed vector database that simplifies deploying vector search into production applications for semantic search and recommendation systems, addressing common barriers such as high costs (reduced by up to 10x through hybrid storage), lack of engineering expertise, and infrastructure complexity; the platform abstracts away the technical details of nearest neighbor algorithms and index management, allowing teams to focus on their core applications while providing features like filtering, SOC2 compliance, and easy integration via REST API.

This section covers infrastructure patterns for deploying vector search at scale. Inference API endpoints abstract model integration, handling retries and buffering while providing unified access to external services like Azure OpenAI. Semantic text fields automate chunking (default 250 words with overlap) and embedding generation, reducing implementation complexity. These abstractions enable developers to focus on application logic rather than infrastructure details, with configurations applied at index creation time and reused across subsequent operations.

Vector search leverages vector embeddings to represent data semantically, enabling machines to work with text, images, audio, and video. Similar items cluster together in vector space, allowing nearest neighbor searches. Vector databases enable this at scale for millions to billions of documents. Pinecone is a fully managed vector database that simplifies production deployment without infrastructure management. It works alongside embedding models to index and query vector data, offering fast, fresh, and filtered search capabilities.

This tutorial demonstrates how to implement vector search on Google Cloud using Vertex AI, covering the complete workflow from reading CSV data to creating text embeddings with the Text-BERT model, ingesting data into a vector database via Google's managed Vector Search service, and performing similarity searches to retrieve relevant candidates based on query prompts. The process involves creating a Vertex AI Workbench environment, generating embeddings from combined text fields, creating an index and deploying it to an endpoint (each taking 35-45 minutes), and finally querying the deployed endpoint to fetch matching results.

First-stage retrieval uses inexpensive lexical search to narrow from millions to hundreds of documents (high recall, low precision). Second-stage retrieval uses expensive but more accurate models to re-rank these candidates (high precision). Advantages of this two-stage approach: (1) Easy to update and A/B test second-stage models without re-embedding all documents; (2) Keeps first-stage cheap and scalable; (3) Ensures sufficient recall before expensive re-ranking. Most deployment focuses on improving second-stage models while keeping first-stage simple.

Modern recommendation systems employ a two-stage pipeline separating retrieval (recall) from ranking (precision). Retrieval narrows millions of items to hundreds/thousands of candidates using fast methods like approximate nearest neighbors or graph traversals. Ranking then applies detailed feature analysis to order these candidates. The offline phase trains embedding models, builds indexes, and constructs feature stores. The online phase uses these artifacts to serve requests efficiently. This separation enables horizontal scaling, where computationally intensive tasks run offline while serving remains lightweight. The architecture balances speed and accuracy—retrieval prioritizes recall, while ranking maximizes precision using richer feature sets including user demographics, contextual signals, and item metadata.

Multi-stage retrieval pipelines combine different methods for efficiency and accuracy: first-stage filtering uses fast methods like BM25 to retrieve many candidates, then second-stage ranking uses stronger neural models to differentiate top results. Neural ranking methods leverage language models for pointwise (individual document evaluation), pairwise (document comparison), or listwise (entire candidate list evaluation) ranking. This two-stage approach balances initial speed with final precision, while neural ranking exploits the generator's ability to understand context and relevance beyond simple keyword matching.

Most real-world applications use a two-stage pipeline balancing speed and accuracy. First stage: bi-encoders quickly scan millions of documents using fast approximate nearest neighbor search, retrieving top 100-1000 candidates (high recall). Second stage: cross-encoders rerank these candidates using joint encoding for high precision. This optimal trade-off provides both the speed of bi-encoders and the accuracy of cross-encoders, making it suitable for semantic search engines and RAG systems.

The ScaNN search process involves two phases: (1) Initial scoring phase: traverse the tree structure, computing distances from the query to root nodes, and select the closest node to traverse down, repeating until the lowest layer is reached. (2) Refinement phase: use the original vectors (not quantized) to compute exact inner products for the top results from the initial phase. This two-phase approach achieves both high speed and high accuracy.

The Pareto frontier consists of solutions that are not dominated by any other solution. In graphical representation, these are the solutions that are not 'oscured' by other points. When making decisions among multiple objectives, one should focus only on the Pareto frontier because all other solutions can be ruled out as inferior. For example, in a trade-off between computation time and solution quality, solutions on the frontier represent the best possible compromises where improving one aspect would necessarily worsen another.

Hyperparameter optimization is a derivative-free problem. Methods include grid search, random search, Nelder-Mead, Simulated Annealing, SMAC, and TPE. Surprisingly, random search often outperforms classical methods, while Bayesian methods like TPE slightly outperform random search. On 30GB image features with 16 nodes: naive grid search takes 800-1000 minutes, while with batching, early stopping, and TPE/random search, completion drops to ~100 minutes. On 1.5TB with 128 nodes, good answers are found within 11 hours versus a week with naive approach.

The ScaNN search process involves two phases: (1) Initial scoring phase: traverse the tree structure, computing distances from the query to root nodes, and select the closest node to traverse down, repeating until the lowest layer is reached. (2) Refinement phase: use the original vectors (not quantized) to compute exact inner products for the top results from the initial phase. This two-phase approach achieves both high speed and high accuracy.

Black box optimization treats hyperparameter tuning as function evaluation where the function is expensive, non-differentiable, and non-convex. Three optimization methods are compared: (1) Random search samples uniformly from the search space and is surprisingly effective because different hyperparameters have different importance levels for different datasets; (2) Gaussian processes model the hyperparameter-quality relationship probabilistically with uncertainty quantification, allowing efficient exploration by focusing on high-uncertainty regions, but have cubic computational complexity; (3) TPE (Tree-structured Parzen Estimator) divides configurations into good/bad sets and models their distributions using kernel density estimation, handling categorical and conditional hyperparameters well but assuming independence between hyperparameters.