LRU Cache Implementation in C++ | LeetCode 146 Explained

This segment covers the implementation of a thread-safe caching mechanism using ConcurrentDictionary. The key concepts include: (1) Using ConcurrentDictionary as a thread-safe collection for caching, (2) Creating a CacheEntry record that stores both the cached value and a creation timestamp, (3) Implementing cache freshness validation by comparing current time against the creation timestamp, (4) Understanding the cache stampede problem where concurrent requests simultaneously miss the cache and all attempt to fetch data from the external source. The segment demonstrates how to structure cache entries with expiration tracking and how to validate cache freshness before returning cached values.

Implementing a cache with mutex: (1) Use sync.Mutex to protect the cache map; (2) Lock before reading/writing to ensure thread safety; (3) For read-heavy workloads, consider using sync.Map which is optimized for concurrent reads; (4) The mutex should be released after the operation completes; (5) Consider using defer to ensure mutex release even if panic occurs.

Simple thread-safe cache implementations contain numerous hidden defects. Separating mutexes from protected data creates semantic disconnection that leads to race conditions. Returning raw pointers breaks ownership semantics and creates exception safety problems. Using const-correctness with mutable mutexes creates subtle bugs that compilers cannot catch. The fundamental problem is that raw mutexes and data have no inherent relationship in the code. The solution is the guarded variable pattern, which ties mutexes directly to the data they protect, eliminating raw mutexes from code and ensuring that any access to the data requires acquiring the associated lock.

Implementing thread-safe caching requires understanding thread safety problems with regular data structures. Regular Dictionary operations are not atomic and can corrupt internal state when accessed concurrently. Using regular locks causes performance issues as readers block each other. The solution involves using ReaderWriterLockSlim: acquire write locks for updates and read locks for queries. Always check lock acquisition success and release in try-finally blocks. This pattern creates a thread-safe global configuration cache allowing multiple readers and unique writers, essential for high-performance web applications handling concurrent user requests.

For multi-threaded scenarios, thread-safe alternatives to Cell, RefCell, and RC are needed. Mutex provides exclusive access with blocking behavior, similar to RefCell but thread-safe. RwLock provides read-write locks with blocking behavior, allowing multiple readers or single writer. Arc (Atomic Reference Counted) is the thread-safe version of RC, using atomic operations for reference counting. These types use CPU-provided synchronization primitives rather than Rust's borrow checker to enforce safety constraints across thread boundaries.

Virtual memory is a memory management technique that allows programs to use more memory than physically available by using a portion of the hard disk as an extension of RAM. The system divides programs into pages and uses a page table with valid/invalid bits to track which pages are in memory versus the backing store. When a page is requested and not found in memory (page fault), the system must evict an existing page to make room. Three main page replacement algorithms exist: FIFO (First-In-First-Out) evicts pages in the order they were loaded but suffers from Belady's anomaly where increasing frames can increase page faults; Optimal evicts the page not used for the longest time in the future, producing the minimum page faults but requiring future knowledge; LRU (Least Recently Used) evicts the page not used for the longest time, based on the principle of locality, and is the most practical for real systems as it doesn't suffer from Belady's anomaly while providing good performance.

Operating systems manage limited, expensive RAM by using virtual memory, which divides memory into fixed-size pages (in secondary memory) and frames (in RAM). When programs need pages not in memory, page faults occur, requiring swap out (evicting pages to secondary memory) and swap in (loading needed pages). Page replacement algorithms determine which page to evict when frames are needed. Key concepts include page hits (pages already in RAM) and page faults (pages requiring loading). Three main algorithms exist: FIFO (evicts oldest page), LRU (evicts least recently used page), and Optimal (evicts page used farthest in future). The hit ratio measures algorithm efficiency.

Virtual Memory Management uses Demand Paging to create an illusion of increased memory. Pages are loaded from secondary memory to RAM only when needed (on demand). When RAM frames are full, existing pages are replaced (swapped out) to make room for new pages. FIFO (First In First Out) evicts the oldest page. LRU (Least Recently Used) evicts the page not used for the longest time. Belady's Anomaly occurs when FIFO with more frames causes more page faults than with fewer frames.

Virtual memory is implemented through demand paging, where pages are loaded from disk only when needed. Page replacement algorithms determine which page to evict when RAM is full. FIFO replaces the oldest page but can suffer from Belady's anomaly. LRU replaces the least recently used page and approximates optimal behavior. The Optimal algorithm replaces the page not used for the longest time in the future, producing minimum page faults but requiring future knowledge.

Virtual memory implementation involves page tables with frame numbers, valid/invalid bits, modified bits, and disk locations. When a page fault occurs, the OS loads pages from disk into free frames and updates tables. Page replacement algorithms determine which page to evict when no free frames exist: FIFO evicts the oldest-loaded page; NRU evicts pages not referenced recently using reference bits; Clock algorithm combines FIFO with reference bits for better performance. These algorithms balance memory utilization, page fault rates, and system performance. The choice of algorithm significantly impacts system efficiency, with FIFO potentially suffering Belady's anomaly.

Distributed caching addresses single-point-of-failure limitations using consistent hashing, which places servers and data on a circular path based on hashed values, enabling efficient horizontal scaling while maintaining cache efficiency. Cache invalidation strategies determine how stale data is removed: Cache Aside Strategy gives applications fine-grained control over cache population but increases complexity; Read Through Strategy abstracts data retrieval to the cache server but risks stale data and incurs overhead for misses; Write Around Strategy avoids cache pollution in write-heavy scenarios but increases read latency; Write Through Strategy ensures consistency through single transactions but adds performance overhead; Write Back Strategy improves write performance through asynchronous updates but introduces data inconsistency risks during the deferral window.

This video explains three fundamental strategies for writing data to distributed caches: write-around (writing to the database first, then optionally invalidating or accepting stale reads), write-through (writing to both cache and database simultaneously for consistency but with slower writes), and write-back (writing to cache first and asynchronously writing to database later for faster writes but with risk of data loss). Each strategy offers different trade-offs between write speed, data consistency, and system complexity, and the optimal choice depends on specific application requirements.

Cache write policies determine how data is updated across caches and databases, balancing consistency, latency, and resource usage. Write Through updates both cache and database simultaneously, ensuring strict consistency but introducing higher latency and resource consumption. Write Back (Write Behind) updates cache first, then database asynchronously, reducing latency but increasing data loss risk during failures. Write Around bypasses cache entirely, updating only the database and invalidating cache entries, minimizing cache pollution for rarely reused data. Refresh Ahead proactively refreshes cache entries before expiration to prevent misses. Distributed caching extends these concepts across multiple networked nodes, providing scalability and fault tolerance. Popular implementations include Memcached and Redis. Cache coherence protocols like MESI (Modified, Exclusive, Shared, Invalid) and MOESI (adding Owned state) ensure data consistency across distributed nodes by managing cache states and preventing stale data usage.

Caching stores frequently accessed data in fast memory to reduce retrieval time. Without caching, each request requires multiple network calls (Client-Server-Database-Server-Client), totaling 8ms. Cache Hit occurs when data exists in cache (3ms total), while Cache Miss requires database fetch (6ms total). Local caching stores data in server memory but causes data inconsistency in distributed systems. Distributed caching stores cache in shared locations (like Redis clusters) accessible by all servers. Cache eviction policies manage limited cache space: FIFO removes oldest data first; LRU evicts least recently used data by tracking recency; LFU evicts least frequently accessed data by tracking access frequency. Cache write policies determine where to write first: Write Through updates both cache and database simultaneously ensuring consistency but slower; Write Back updates cache first then database asynchronously, faster but may cause temporary inconsistency.

Distributed caching uses centralized stores like Redis to serve multiple application instances. The fundamental challenge is maintaining cache synchronization with persistent data stores. Two primary strategies address this: write-through caches perform synchronous writes to both cache and disk, ensuring immediate consistency but higher latency. Write-back caches perform asynchronous writes, improving performance but introducing complexity around data durability and potential data loss if the cache fails before flushing changes.