Buffer Pool Management in Database Systems | CMU 15-445

Added:

Memory-Mapped I/O (mmap) Database Architecture

Traditional database systems education strongly advocates for user-space buffer pool management, arguing that operating system paging lacks database-specific semantic knowledge. However, a significant counter-perspective advocates for Memory-Mapped I/O (mmap) to let the OS handle page caching. Proponents of mmap-based architectures (used in engines like LMDB) argue that delegating memory management to the OS kernel drastically simplifies database codebase complexity, reduces user-space overhead, and leverages highly optimized kernel-level page tables. By mapping files directly into the virtual address space, the DBMS treats storage as virtual memory, bypassing the need for a complex, custom user-space buffer pool manager.

Understanding of the computer memory hierarchy, specifically the performance and latency differences between volatile RAM and non-volatile disk storage.
Basic concepts of Operating Systems memory management, including virtual memory, physical memory pages, and OS-level page replacement.
Knowledge of disk-oriented database storage structures, specifically how tables and indexes are mapped to fixed-size disk pages.
Familiarity with basic data structures, particularly Hash Tables and Doubly Linked Lists, which are essential for tracking pages in-memory.
Implementation of concurrency control within the buffer pool, including page latching (Read/Write locks) and tracking pin/reference counts to prevent active page eviction.
Advanced page replacement algorithms optimized for database workloads, such as LRU-K, 2Q, and CLOCK-Pro, to mitigate the sequential scan pollution problem.
Buffer pool optimization strategies, including sequential pre-fetching, scan sharing, and buffer pool bypass (direct I/O) for large bulk operations.
The integration of buffer pool state with Database Recovery mechanisms, specifically Write-Ahead Logging (WAL) and the STEAL/NO-FORCE buffer management policies.
43.8K views577likes1:18:59@CMUDatabaseGroupOriginal Release: 2019-09-12

A buffer pool is a memory cache that manages disk page access in database systems, using a page table to map page IDs to memory frames and employing replacement policies like LRU or Clock to decide which pages to evict when memory is full, with optimizations including multiple buffer pools, prefetching, scan sharing, and buffer pool bypass to minimize disk I/O and improve query performance.