Building a B+Tree Database: Node Binary Representation Explained

Multi-byte values are stored in memory as individual bytes. For example, the number 500 (binary: 111110100) is stored as two bytes: 0xCC (244) in the low-order byte and 0x01 in the high-order byte. Endianness determines whether the low-order byte comes first (Little Endian, used by x86/x64) or the high-order byte comes first (Big Endian, used by PowerPC). This is an arbitrary design choice, not a physical law.

Endianness refers to how multi-byte values are stored in memory. Big Endian stores the most significant byte at the lowest memory address (like writing numbers left-to-right). Little Endian stores the least significant byte at the lowest memory address (like writing numbers right-to-left). The name comes from Gulliver's Travels, where the Lilliputians broke eggs from the small end. This affects how binary files and network protocols are interpreted. Bitwise operations work on individual bits: AND (&) returns 1 only if both bits are 1; OR (|) returns 1 if either bit is 1; XOR (^) returns 1 if bits differ. XOR has special properties: A XOR A = 0, A XOR 0 = A, and A XOR B XOR A = B. These operations are essential for bit manipulation, flag handling, and cryptographic algorithms.

Six primary bitwise operations manipulate individual bits within words: AND (logical conjunction), OR (logical disjunction), XOR (exclusive or), NOT (inversion), and shifts (left/right movement). Each operation applies logic to corresponding bit pairs or single bits. Shifts move all bits while optionally filling vacated positions with zeros, ones, or rotated bits. These operations enable precise control over binary data representation and manipulation.

All computer data is fundamentally composed of zeros and ones, stored in 8-bit bytes. The Format-Hex command reveals binary data in hexadecimal format for readability. Hexadecimal uses digits 0-9 and letters A-F to represent 16 values, making binary more compact. Character encoding standards like ASCII and UTF-8 map characters to numerical values, with ASCII using 1 byte per character and UTF-8 using variable-length encoding. Data types like Int32 (4 bytes) and Int64 (8 bytes) determine storage requirements. The little endian convention stores bytes from least to most significant, so 1234 appears as 00 04 D2. Negative numbers use two's complement representation, and floating point numbers follow the same byte ordering.

Endianness refers to the byte ordering scheme used by computers to store multi-byte data in memory. This concept is essential for binary exploitation because when manually writing data into memory (such as during buffer overflow attacks), one must account for how bytes are arranged. Functions like p32/p64 from pwntools or Python's struct module are used to pack and unpack bytes with specified endianness. Understanding endianness is crucial because it determines how the CPU interprets bytes in memory, and incorrect byte ordering will cause the CPU to misread values entirely.

Structs are allocated in the Stack segment with members taking specific sizes (char: 1 byte, int: 4 bytes). Total size includes padding for alignment. Most compilers allocate members in declaration order, but C++ standard doesn't require this. The dot operator (.) accesses members of struct objects, while the arrow operator (->) accesses members through pointers. The address-of operator (&) and dereference operator (*) work identically with structs as with basic types. Struct members can hold pointers to other structs, enabling dynamic linking. Memory allocation uses new for structs and arrays, with delete for freeing. Always check for NULL before using or deleting allocated memory.

Structs in C store items of different types in contiguous memory with fixed size, unlike arrays of same-type items. Define structs using 'struct' keyword with fields of various types. Use malloc() for heap allocation, where size depends on field types (char pointer = 8 bytes, float = 4 bytes). Every malloc requires a free() to prevent leaks. When functions return pointers, determine ownership: make_goat returns owning pointers (caller must free), while fattest_goat returns non-owning pointers (caller doesn't free). This ownership model is essential for preventing memory leaks and double-free errors in manual memory management.

This comprehensive section covers memory addressing, pointers, and struct fundamentals. When variables are passed to functions by value, only copies are created, so changes inside the function don't affect original variables. To modify original variables, we need memory addresses. A pointer is a variable that stores memory addresses rather than data values. Memory addresses are dynamic values that change each program run, representing physical RAM locations. The address-of operator (&) retrieves a variable's address, while the dereference operator (*) accesses the value at that address. Pointers are declared using the asterisk (*) symbol. Pointers can be assigned addresses, incremented/decremented for sequential access, and used to modify original variables. Only subtraction between pointers has meaningful results (calculating distance between memory locations), while addition, multiplication, and division have no practical meaning. The size of a pointer depends on system architecture (32-bit or 64-bit), not on the data type it points to. Memory alignment refers to how data is organized in memory for efficient access. The compiler adds padding bytes to ensure data types are aligned at specific memory boundaries. A struct (structure) is a composite data type that groups multiple variables of different data types under a single name. Struct fields are accessed using the dot operator (.) after the struct variable name. Structs can be assigned to each other, copying all fields simultaneously. Structs can also contain other structs, pointers, or arrays as fields.

Pointers are variables storing memory addresses of other variables, declared with the * operator. The address-of operator (&) assigns addresses to pointers, while the dereference operator (*) accesses values at those addresses. Pass-by-reference using pointers allows functions to modify original variables, unlike pass-by-value which only copies values. Pointer arithmetic moves pointers by element count rather than bytes, with ptr + 1 advancing to the next element of the same type. Dynamic memory allocation uses malloc() (allocates without initialization), calloc() (allocates and initializes to zero), and free() (releases memory). After allocation, checking for NULL detects failures. Structs group multiple variables of different types under a single name, useful for organizing related data. Unions share memory among all members, with size determined by the largest member, allowing only one member to be used at a time.

C++ requires manual memory management where developers explicitly allocate and deallocate memory. This involves understanding pointers as 64-bit memory addresses, using malloc for raw memory allocation, and casting void pointers to appropriate types. Pointers automatically increment by their data type's size (e.g., 24 bytes for a 24-byte object), enabling array traversal without manual offset calculations.

A data block is the minimum amount of data that can be read or written by a single I/O operation, typically ranging from 4KB to 32KB with 8KB being most common. Unlike data pages which DBMS manages, data blocks are managed by the underlying storage system (disk or SSD). DBMS maintains a mapping between data pages and data blocks because it has no control over where physical data blocks are located on disk. One data block can store one or multiple data pages depending on the block size relative to page size.

Databases store data in fixed-size pages (typically 8KB) on disk. When reading, the database retrieves entire pages rather than individual records. This page-based approach optimizes disk I/O by minimizing read operations. Pages are organized sequentially, with each page containing multiple records. When records are deleted, free space is created within pages, requiring careful management. The buffer pool stores recently accessed pages in memory, reducing disk reads. Understanding this architecture is fundamental to database performance optimization.

Deadlocks occur when two or more transactions wait indefinitely for each other to release locks. For example, if transaction T1 holds lock on resource X and waits for Y, while T2 holds lock on Y and waits for X, neither can proceed. Deadlock detection uses wait-for graphs where nodes represent transactions and edges represent wait relationships. When a cycle is detected, one transaction must be aborted to resolve the deadlock. Multi-Version Concurrency Control (MVCC) is an alternative to locking that improves concurrency by maintaining multiple versions of data. When a transaction reads data, it sees a consistent snapshot of the database as of its start time. When a transaction writes data, a new version is created. This allows readers to proceed without blocking writers and vice versa. Databases store data in fixed-size pages (typically 8KB) on disk. When a page is modified, it becomes 'dirty' - meaning it differs from the version on disk. This abstraction simplifies concurrency control because the database can lock entire pages rather than individual records.

The presenter explains that databases store data in files, with each file representing a table or similar structure. At the physical level, storage systems operate at 4KB blocks, meaning any data read or write must be aligned to 4KB boundaries. Databases use 'pages' as their fundamental storage unit, with each page being 4KB in size. The heap is the file structure that stores table data, containing all records organized for efficient access. The disk manager is responsible for reading and writing pages to and from disk, abstracting away the complexity of direct file I/O. The presenter explains that page size is a fixed value (typically 4KB) that databases use for all storage operations, determined by the underlying storage hardware.

The fundamental unit of database operations is the page or block (typically 8KB), not individual records. Data is stored on these pages, and accessing data requires reading the entire page. Database cache stores frequently accessed data in memory for faster access, reducing disk reads. Both read and write operations require loading pages into memory first. This page-level operation model is essential for understanding database performance and why cache is critical.