OS Development Tutorial: Entering Protected Mode & GDT

Paging is a memory management scheme that extends physical memory using secondary storage as virtual RAM. It divides both logical and physical address spaces into fixed-size pages and frames of equal size. The Memory Management Unit translates logical addresses (page number + offset) to physical addresses using page tables stored in main memory. Each process maintains its own page table mapping virtual pages to physical frames. When a page is not in RAM, a page fault occurs, triggering the OS to load it from disk. This enables non-contiguous memory allocation and allows programs to use more memory than physically available, solving limitations of physical memory size in modern computing systems.

Virtual memory works by temporarily swapping parts of program data and code between RAM and a dedicated area of the hard disk called the page file. The process of swapping pages between RAM and disk is called paging. Virtual memory allows programs to require more memory than physical memory is available, as only the demanded areas of programs are kept in physical memory at any given time.

Paging divides both programs and memory into fixed-size pages and frames, eliminating external fragmentation. Programs are divided into pages (typically 512-64 KB), and memory is divided into frames of equal size. Pages can be stored in non-contiguous frames. The OS maintains page tables mapping page numbers to frame numbers. Virtual addresses consist of page numbers and offsets; physical addresses are formed by replacing page numbers with frame numbers. Virtual memory provides an illusion of more memory than physically available, enabling execution of programs larger than physical memory. It relies on spatial locality (programs access nearby memory) and temporal locality (recently accessed locations are likely accessed again soon).

Virtual memory solves three major problems with direct RAM access: (1) Security vulnerabilities where programs can interfere with each other's memory, (2) Fragmentation where contiguous memory allocation creates unusable gaps, and (3) Insufficient memory when programs exceed available RAM. The solution involves dividing memory into fixed-size blocks called pages and frames, enabling the operating system to map virtual addresses to physical memory or disk storage. This approach eliminates external fragmentation, allows programs to exceed physical RAM capacity through swapping, enables shared libraries to be loaded once for multiple processes, and supports demand paging to load only necessary program components initially.

Virtual memory creates illusion of larger memory than physically available. Logical memory differs from physical memory. Logical addresses are translated to physical addresses using relocation registers. Page size equals frame size and is fixed. Logical address space is divided into pages, physical address space into frames. Demand paging loads pages on demand from virtual memory to physical memory.

The transition from assembly to C code involves nucking the frame pointer (EBP) and setting the stack pointer (ESP). The assembly code prepares the stack by pushing EBP and setting ESP to point to the new stack frame. This is the first stack frame of the kernel, as prior to this, the code was running on the bootloader's stack frame. The C code then continues with normal function prologue operations, including pushing EBP and setting up the new frame.

The bootloader transitions control to C code by jumping to an assembly entry point that initializes the CPU state. This involves clearing AX, setting SS and SP registers to appropriate values, initializing BP, enabling interrupts with STI, and then calling the C entry point (_start). After C code completes, execution halts with CLI and HLT instructions to prevent further processing. This bridge between pure assembly and C code enables using higher-level language features while maintaining direct hardware control.

The pre-kernel bootstrap code must be written in assembly rather than C because the CPU boots directly into 64-bit long mode without passing through 16-bit real mode. The pre-kernel sets up initial paging structures and maps the higher-half kernel into memory before the C runtime can execute. If the pre-kernel were written in C, the linker would place it at a low address, but the kernel expects to execute at a very high address. Without proper higher-half mapping, the kernel would attempt to execute at an invalid address, causing immediate crashes. The pre-kernel must install page tables that map the high virtual addresses to actual physical memory locations before jumping to the higher-half kernel entry point.

Before executing C code, a stack must be established. The stack pointer (ESP) is set to a reserved memory region (typically 0x7c00 + offset), growing downward. This stack enables function calls and local variable storage required by C runtime. The boot loader then switches to C code, which reads the ELF kernel from disk. The ELF format requires verifying the magic number to confirm it's a valid executable. This transition from assembly to C represents a critical shift toward higher-level abstractions while maintaining low-level control over hardware initialization.

This segment advances text rendering capabilities toward hardware-level control. Topics include: (1) Implementing proper CR/LF handling with cursor position updates; (2) VGA text mode memory layout with 160 bytes per line (80 characters × 2 bytes); (3) Calculating cursor bytes by multiplying row by 80 and adding column; (4) Using CRT microcontroller ports (0x3D4 index, 0x3D5 data) for hardware cursor positioning; (5) Writing low byte (0x0F) then high byte (0x0E) to cursor registers; (6) Debugging relative jump range issues in large assembly files. This represents the transition from software-only rendering to direct hardware manipulation for more responsive cursor control.

This section covers hardware communication methods and VGA text mode screen implementation. The instructor explains memory-mapped I/O (writing to specific memory addresses like 0xB8000 for video memory) and port I/O (for devices like keyboards and mice). The system.h file provides C interfaces (importB and outputB functions) to hardware ports. The VGA text mode screen is structured as a table with 80 columns and 25 rows, where each cell stores 2 bytes (character and color attribute). The clear_line function clears specific line ranges by setting character and color bytes to zero. The clear_screen function clears the entire screen by calling clear_line with parameters from line 0 to line 24. The cursor_update function calculates the memory address for cursor position using the formula: cursor_y * screen_width + cursor_x, which is sent to hardware ports (0x3D4 and 0x3D5) to position the cursor.

Hardware scrolling is implemented in VGA systems by using programmable registers (implemented with latch chips like 74LS574) to control the parallel load inputs of horizontal and vertical counters, allowing smooth image movement without redrawing the entire frame; this is achieved through memory-mapped I/O where specific memory addresses correspond to register locations, enabling software to dynamically update scroll positions during the vertical blanking interval for efficient animation.

The fbcon text console driver implements a sliding window over a buffer of text and attribute bytes, reusing code from VGA text mode. When text is written or scrolled, the framebuffer representation must be updated. Performance optimizations include kernel-level hardware acceleration for character rendering (expanding bitmapped fonts to use foreground/background colors), rectangle copying/clearing for scrolling, and panning (moving a window over memory). Hardware wrapping allows scrolling without moving data by changing hardware registers. Smart redraw optimizes by skipping redraw of identical characters overlapping during scroll operations. The choice of optimization depends on available hardware features.

Ring 3 (user mode) represents the lowest privilege level and contains user applications like Google, Discord, and games. Programs in this ring are constantly monitored and cannot directly access other programs' memory or communicate directly with hardware components such as graphics cards and keyboards.

Multitasking is the ability of a computer to perform multiple tasks simultaneously. In reality, the CPU rapidly switches between different programs, creating the illusion of simultaneous execution. A task is a single job or specific work unit that a program performs. The 80386 supports task management through specific data structures: Task Register (TR) holds the address of the current Task State Segment, Task State Segment (TSS) stores task context, Task State Segment Descriptor (TSSD) defines TSS properties, Task Gate Descriptor (TGD) provides indirect reference to TSS, and Task Linking connects tasks in a chain. The 80386 supports four privilege levels (rings 0-3) in protected mode, with Ring 0 having the highest privilege and Ring 3 having the lowest.

Multitasking enables the 80386 to execute multiple tasks simultaneously through time-sharing, rapidly switching between applications. Task switching occurs via interrupts, exceptions, or jump instructions. The Task State Segment (TSS) is a 104-byte data structure (26 locations × 4 bytes) that saves comprehensive task state information, including all 32-bit registers, 16-bit segment selectors, CR3 (page directory address), and LDT pointer. The TSS contains dynamic fields (EAX, EBX, ECX, EDX, ESI, EDI, ESP, EBP, CS, EIP) that change with each switch, and static fields (I/O permission bitmap, T-bit, PDB) that remain constant. The 80386 has four privilege levels: Level 0 (OS kernel), Level 1 (OS programs), Level 2 (device drivers), and Level 3 (application software). The CPL (Current Privilege Level) is stored in the CS register's lower two bits. The DPL (Descriptor Privilege Level) is in the segment descriptor, and RPL (Requested Privilege Level) is in the selector. The TSS descriptor is a 64-bit structure with a 32-bit base address field, 20-bit limit field, G bit (granularity), P bit (present), DPL field, and B bit (busy). The task register (TR) contains a selector field pointing to the TSS descriptor in the GDT. The task register can be programmed using LTR (Load Task Register, privileged) and STR (Store Task Register, non-privileged) instructions.

The Task Register (TR) is a 16-bit register that stores a selector pointing to the Task State Segment (TSS) descriptor in the GDT. Each task in the 80386 microprocessor has its own Task Register and Task State Segment. The Task Register is used during multitasking to switch between tasks. When a task switches, the CPU uses the Task Register to access the corresponding TSS. The Task State Segment (TSS) is a special memory segment that stores all the necessary information required to restart a task where it was left off. When one task switches to another, the TSS stores the complete state of the current task, including CPU register values, stack pointer, and other execution context information. This allows the CPU to resume the task exactly where it was left off when it is scheduled to run again. The TSS is addressed using the Task State Segment Descriptor, which contains the base address, limit, and access rights. The TSS descriptor is located in the GDT. The TSS contains a linkage field that provides a backlink to the previous task, enabling the nesting of tasks and allowing a task to be interrupted by another task and then resume its execution.

This section covers task state segment (TSS) and virtual mode in 80386. The task register is a 16-bit register storing the TSS selector for multitasking. TSS contains base addresses and limits for code, data, and stack segments, task link, and I/O map base address. The minimum TSS size is 104 bytes. Virtual mode (Protected Mode) differs from real mode in providing memory protection, requiring VM bit to be set to 1. Memory management in Virtual 8086 Mode involves 1 MB total memory, supporting multiple Virtual 8086 instances with hardware and software combination for memory management. Entering and leaving virtual mode occurs through task switches, interrupt handling, or privileged instruction execution. Task switching involves saving current task state to its TSS, setting busy bit in new task descriptor, loading new task's TSS into task register, and resuming execution. Task switching can occur with or without task gate, with different privilege checking requirements.