Inverted Index Explained | Information Retrieval | Stanford NLP

Postings lists in inverted indexes store document identifiers for each term and typically occupy significantly more space than the dictionary—often 10 times larger or more. For the RCv1 corpus with 800,000 documents, doc IDs range from 1 to 800,000, requiring approximately 20 bits per doc ID using the formula floor(log₂(n)) + 1. A fundamental trade-off exists between common and rare terms: common terms like 'the' appearing in many documents waste space with 20-bit encoding, while rare terms with few postings can accept 20 bits. Delta encoding addresses this by storing differences between consecutive doc IDs rather than absolute values, using fewer bits for common terms with small gaps and more bits for rare terms with larger gaps. Variable length encoding uses fewer bits for smaller values and more bits for larger values, achieving approximately log₂(G) bits for gap size G. Variable byte encoding uses continuation bits (first bit of each byte) to indicate whether more bytes follow: 1 means last byte, 0 means more bytes follow. Values ≤127 fit in one byte; larger values use multiple bytes. This encoding is unambiguous and aligns with computer architecture. Gamma code represents G as (length, offset) where offset is binary G with MSB removed, and length is unary representation of offset length. Gamma code has exactly 2*log₂(G) + 1 bits and is parameter-free, adapting to any corpus without knowing document count in advance.

Gamma coding is a bit-level compression technique that stores a pair of length and offset. For a value like 13, it encodes the binary representation (1101), calculates the length (3 bits), and combines them using unary coding for the length. This technique reduces memory requirements by a factor of two compared to variable length encoding while maintaining unique prefix decodability. The video concludes by comparing all compression techniques (dictionary, posting with gap compression, variable length encoding, and gamma coding) in a summary table, showing how each technique reduces memory requirements differently based on term frequency and document distribution patterns.

Variable byte encoding works by repeatedly dividing a number by 128 (2^7) and recording remainders. Each remainder is stored as a 7-bit value, with the continuation bit indicating if more bytes follow. For example, 214577 is decomposed as follows: 214577 ÷ 128 = 1676 remainder 49; 1676 ÷ 128 = 13 remainder 12; 13 ÷ 128 = 0 remainder 13. These remainders (13, 12, 49) are then encoded in sequence with continuation bits, allowing efficient representation of large numbers using only as many bytes as necessary.

Variable Byte Encoding stores compressed posting lists in binary format. Each gap value is divided into 7-bit chunks with a continuation bit (1 bit) following each chunk. A continuation bit of 0 indicates the next chunk belongs to the same delta value, while 1 indicates a new delta value. This method is also called 'byte aligned code' because each chunk plus its continuation bit equals exactly one byte.

Compression serves multiple purposes: reduces storage requirements, improves cache utilization, and enables faster network transfer. Modern compression algorithms like LZ77 and Snappy achieve speeds of hundreds of MB/s to GB/s, which is faster than disk read speeds (typically 100MB/s for SATA). Compression in search systems can be lossless (no information loss) or lossy (some information discarded). Lossless compression is used for most index data because exact reconstruction is required. Lossy compression is used when some information can be safely discarded, such as converting text to lowercase or approximating term frequencies. Delta encoding stores differences between consecutive document IDs instead of absolute values. Variable-length byte encoding (Varint) is a byte-based encoding scheme where each number is encoded in one or more bytes. The first byte contains the value if it's less than 128, or the value modulo 128 plus a continuation bit (1) if more bytes are needed. Bit-level encoding achieves better compression ratios but has higher computational complexity for decoding. Fibonacci coding is a bit-based encoding scheme that represents numbers using Fibonacci numbers. The encoding appends a '1' bit to indicate the end of the code. Gamma coding is a bit-based encoding scheme that encodes the length of the binary representation of a number, followed by the binary representation itself. The length is encoded as the number of leading zeros plus one. Rice coding is a parameterized encoding scheme that combines unary encoding with binary encoding. The algorithm: (1) Choose a parameter k (typically a power of 2), (2) Encode the number as k unary bits followed by the binary representation of the remainder. Rice coding is adaptive and can be tuned to the specific distribution of deltas in the posting list. Simple9 is a word-aligned encoding scheme that combines multiple encoding strategies. It uses 4 bits to specify which encoding scheme to use for the following data, followed by the encoded data. The 4 bits allow for 16 different encoding schemes, each optimized for different patterns of deltas. The decoder uses a jump table to quickly access the appropriate decoding routine for each scheme. Word-aligned encoding (like Simple9) uses fixed-size words (typically 32 bits) for encoding, which is faster to decode than bit-level encoding. However, it may achieve lower compression ratios because it cannot pack data as tightly as bit-level encoding.

Positional indexes extend traditional inverted indexes by storing not just document IDs but also the specific positions where terms appear within each document, enabling efficient handling of phrase queries (multi-word units like 'Stanford University') and proximity queries (words within a specified distance of each other). While byword indexes (indexing consecutive word pairs) offer an alternative approach, positional indexes are the standard solution despite requiring 2-4 times more storage space than non-positional indexes, as they provide the flexibility to handle both explicit phrase queries and implicit phrase queries while supporting proximity-based search capabilities.

A positional index stores the positions of terms within documents, enabling efficient phrase query processing by using a two-pointer traversal algorithm where one pointer stays exactly one step behind the other to verify that consecutive terms appear adjacent in the document; this approach can also be adapted for proximity queries by allowing a configurable distance K between term positions.

Positional indexes store the exact positions (offsets) of terms within documents, enabling efficient phrase query processing by checking if multiple terms appear consecutively or within specific proximity ranges; unlike standard inverted indexes that only record document-term relationships, positional indexes allow systems to verify that query terms appear together in the same sequence within documents, though they require significantly more storage (typically 2-4 times larger than standard inverted indexes).

This lecture covers two key topics in information retrieval: (1) Phrase Queries - methods for searching complete phrases with exact word order, including Biword Indexes (which store consecutive word pairs but suffer from false positives) and Positional Indexes (which store exact word positions within documents, enabling precise phrase matching); (2) Query Optimization - the strategy of executing AND/OR queries by processing the smallest posting lists first to minimize computational cost, improving search performance.

Proximity indices enable efficient phrase searching by storing positional information of terms in documents rather than just frequencies; unlike traditional inverted indices, they maintain lists of (document, position) tuples for each keyword, allowing the linear merge algorithm to check for document ID equality and verify that terms occur at adjacent or specified proximity positions to identify matching phrases.

BM25 satisfies three natural properties making it a robust ranking function: (1) Zero score for absent terms, (2) Increasing score with term frequency, (3) Bounded maximum value as frequency increases. When b=0, length is ignored; when k→∞, BM25 reduces to simple TF-IDF. Implementation requires: computing term frequencies during inverted index construction, storing document lengths, calculating IDF as log₂(N/DF), and applying the BM25 formula during query processing. This involves two passes: first building the inverted index with TF counts, then computing BM25 scores using document length and average document length statistics.

Query processing uses set intersection on the inverted index for multi-term queries. TF-IDF (Term Frequency-Inverse Document Frequency) measures document relevance by combining term frequency (how well a term describes a document) with inverse document frequency (how rare a term is). BM25 is the default Lucene/Solr/Elasticsearch algorithm, a modified TF-IDF with additional parameters. The algorithm scores each document by combining term frequency and inverse document frequency scores for all query terms, then ranks documents by total score.

Term-based retrieval uses keywords to find relevant documents. Term Frequency (TF) measures term occurrences in documents, while Inverse Document Frequency (IDF) measures term importance inversely to document frequency, calculated as IDF(T) = log(N/CT). TF-IDF combines these metrics to score document relevance. ElasticSearch (2010) uses inverted indexes mapping terms to documents. BM25, developed in the 1980s, improves TF-IDF by normalizing term frequency by document length. These approaches work well out of the box but have limited tunability for performance improvement.

Classical information retrieval systems use mathematical approaches to determine document relevance to queries. The term-document matrix records word frequencies across documents, forming the foundation for these methods. TF-IDF (Term Frequency-Inverse Document Frequency) calculates relevance by multiplying term frequency (how often a word appears in a document) with inverse document frequency (how rare a word is across the corpus), rewarding terms that are frequent in specific documents but rare overall. BM25 (Best Match 25) enhances TF-IDF by introducing hyperparameters K and B: K controls how much scoring flattens at high frequencies, and B adjusts the penalty for long documents relative to average length. Both approaches rely on inverted indexes that map terms to documents with pre-computed scores for efficient query processing. These classical methods remain important components in modern neural IR systems, often serving as fast baseline models before neural re-rankers refine results.

The B parameter controls document length importance—setting B=0 removes length consideration entirely, making long and short documents with the same term occurrences receive identical scores. The K1 parameter controls term saturation rate—lower K1 causes faster saturation (first occurrence contributes most), while higher K1 slows saturation, giving more advantage to multiple occurrences. Setting K1=0 makes terms saturate immediately, eliminating the benefit of multiple occurrences. BM25 improves upon TF-IDF by adding document length normalization (B) and tunable term saturation (K1). TF-IDF accounts for term frequency and document frequency but lacks these controls. Lucene's practical scoring function (pre-BM25) already incorporated many BM25 features.

This lecture explains how modern search engines handle web-scale indexing through distributed computing frameworks like MapReduce, which enables parallel processing across commodity machine clusters to manage the massive data volumes of the World Wide Web. The MapReduce paradigm transforms complex indexing tasks into simple map and reduce operations that can be executed in parallel, with the master-slave architecture coordinating workloads across distributed nodes. Additionally, the lecture covers dynamic indexing strategies including logarithmic merging, which amortizes the computational cost of merging indexes over time by maintaining a series of indexes where each subsequent index is twice as large as the previous one, thereby improving query performance while handling evolving document collections.

Building large-scale search indexes requires distributing document collections across multiple mappers. Each mapper tokenizes its assigned documents and emits key-value pairs where keys are terms and values are document ID-position tuples. The shuffle phase hashes keys and uses modulo operations to route identical keys to the same reducer. Reducers then merge partially sorted lists from different mappers to construct complete inverted lists for their assigned terms. This parallel approach enables efficient indexing of massive document collections that cannot fit on a single machine.

Building scalable search engines requires addressing storage and distribution challenges. The indexing process involves text analysis, creating term-document pairs, sorting, and writing to disk. For large datasets, divide and conquer processes documents in batches, writing local posting lists that are later merged using merge sort. This approach enables MapReduce-style distributed indexing. For indexes too large for memory, sharding partitions data across servers. Document partitioning (each server handles document subsets) is preferred over term partitioning because query result merging requires only a simple union rather than complex intersections. IDF computation in distributed systems requires background processes to aggregate global statistics and redistribute values.

This section covers the master-worker architecture used by modern web search engines like Google. The master machine coordinates worker machines that perform two core tasks: parsing documents into normalized term-document ID pairs and inverting these pairs into an inverted index. A forward index lists terms per document, while an inverted index lists documents per term—this reversal enables efficient search. The system handles massive data volumes by splitting corpora into manageable chunks called splits, similar to block processing in external sorting. Worker machines dynamically switch between parsing and inversion roles based on availability and workload, demonstrating the flexible resource management essential for large-scale distributed systems.

MapReduce is Google's framework for distributed data processing with Map and Reduce phases. The Map phase converts input documents into term-document ID pairs (key-value pairs). The Reduce phase aggregates these pairs to build the inverted index. For example, term 'A' appearing in documents 1, 2, and 5 becomes (A, [1, 2, 5]). The framework handles distribution, fault tolerance, and parallel execution. Document frequency counts documents containing a term, while term frequency counts occurrences within a document. These metrics are essential for search engine ranking. The master node assigns partitions to workers, enabling efficient processing of large datasets. This architecture allows search engines to handle billions of documents through parallel execution and fault-tolerant design.