Building a Rust Debugger: ptrace, ELF & DWARF

Learning Goal: Designing and implementing a command-line debugger from scratch in Rust using the Linux ptrace system call, ELF file parsing, and the DWARF debugging format to master low-level process control and binary instrumentation.

Estimated Total Study Time: 38 Hours
Prerequisites: Intermediate Rust familiarity (ownership, lifetimes, pattern matching) and basic Linux command-line/systems administration awareness.

Module 1: Systems Programming Foundations & Rust

This module establishes the foundational knowledge required for systems-level development. You will explore how Linux handles processes, transition from user mode to kernel mode via system calls, and examine how Rust serves as a safe, high-performance alternative to C for low-level tasks. Special emphasis is placed on introducing low-level crates like nix and managing resources without a garbage collector.

Knowledge Checkpoint

Differentiate between a standard library function call (e.g., standard formatting) and a raw system call.
Explain how the CPU switches privileges from ring 3 (User Space) to ring 0 (Kernel Space) using interrupts or syscall instructions.
Write a simple Rust program that calls standard UNIX APIs via the safe abstraction layers of the nix crate instead of linking raw C libc elements directly.

Module 2: Process Control and the Linux Ptrace API

In this module, you will master the ptrace (process trace) system call, the foundation of modern Linux debuggers. You will learn to establish parent-child relationships, launch child targets with tracing enabled, wait for state changes, and read or write memory bytes within another process's virtual address space.

Gap Mitigation: Using the `nix` Crate for Safe `ptrace`

To build a debugger in Rust, you should avoid unsafe direct system calls by using the nix crate's ptrace module (nix::sys::ptrace and nix::sys::wait). The pattern involves:

Forking the process with fork().
In the child process: execute ptrace::traceme() followed by an exec variant to load the target program.
In the parent process: monitor execution using waitpid(...).

Knowledge Checkpoint

Explain why the child must execute PTRACE_TRACEME before loading the target binary with execve.
Implement a safe fork in Rust where the parent waits for the child to stop using nix::sys::wait::waitpid.
Describe the difference between PTRACE_CONT, PTRACE_SINGLESTEP, and PTRACE_PEEKDATA.

Module 3: Software Breakpoints and CPU Registers

This module covers the core mechanics of software breakpoints. You will learn how to intercept execution by modifying the target process's memory. This involves replacing a valid CPU instruction with the x86-64 trap byte 0xCC (the INT 3 instruction), catching the resulting SIGTRAP, reading and updating the instruction pointer (RIP) register to point to the original location, restoring the original byte, and single-stepping to resume normal execution.

Gap Mitigation: Register Manipulation in Rust with `nix`

To modify thread registers (specifically the Instruction Pointer RIP) after hitting an INT 3 breakpoint:

Use ptrace::getregs(pid) to retrieve a struct of target register values.
When an INT 3 is triggered, the CPU executes 0xCC and increments RIP to breakpoint_address + 1.
To resume execution properly, update the register struct by subtracting 1 from regs.rip.
Restore the original instruction byte at breakpoint_address using ptrace::write.
Call ptrace::setregs(pid, &regs) to write the updated register state back to the CPU.
Single-step using ptrace::step(pid) to execute the original instruction, re-apply the breakpoint, and then call ptrace::cont to resume execution.

// Conceptual snippet using nix to backup RIP register let mut regs = ptrace::getregs(child_pid).expect("Failed to get registers"); regs.rip -= 1; // Backstep RIP to point at the original instruction address ptrace::setregs(child_pid, regs).expect("Failed to set registers");

Knowledge Checkpoint

Explain why the instruction pointer RIP is incremented by 1 byte after hit-testing an INT 3 instruction.
Implement a function that reads 8 bytes from a virtual address, replaces the first byte with 0xCC, writes the patched value back, and stores the original byte for restoration.
Describe the sequence of steps needed to execute the original instruction at a breakpoint without missing future hits of that same breakpoint.

Module 4: ELF Layout: Inside Linux Executables

Before automating address lookup, you must understand the file layout of Linux executables. This module covers the structure of the Executable and Linkable Format (ELF), explaining headers, program segment views used by the OS loader, and section table views used by linkers and compilers. You will learn to use the goblin crate to parse headers, symbol tables, and section offsets.

Gap Mitigation: Programmatic ELF Parsing in Rust

To read symbolic debugging targets natively, write a helper file to parse ELF files using the goblin crate.

use std::fs::File; use std::io::Read; use std::path::Path;

pub fn find_elf_symbols(path: &Path) -> Result<(), goblin::error::Error> { let mut buffer = Vec::new(); File::open(path)?.read_to_end(&mut buffer)?;

// Parse target binary buffer
let elf = goblin::elf::Elf::parse(&buffer)?;

// Iterate over the symtab (Symbol Table)
for sym in elf.syms.iter() {
    let name = elf.strtab.get_at(sym.st_name).unwrap_or("<unknown>");
    if sym.is_function() {
        println!("Function: {} at address {:#x}", name, sym.st_value);
    }
}
Ok(())

}

Knowledge Checkpoint

Explain the difference between Program Headers (Segments) and Section Headers in an ELF binary. Which one is used by the operating system kernel to load a process?
Find the purpose of the .strtab and .symtab sections in an ELF file.
Write a Rust function using goblin that opens a local file, reads its header, and returns the entry point address.

Module 5: DWARF Debugging Metadata Demystified

This module introduces DWARF, the debugging format that maps raw instruction addresses to source code files, line numbers, and variable names. You will study DWARF concepts such as Debugging Information Entries (DIEs), Compilation Units (CUs), and the DWARF state machine (Line Table). You will learn how to extract this information using the gimli crate in Rust.

Gap Mitigation: Using the `gimli` Crate to Parse DWARF

Because online video tutorials on the gimli crate are rare, here is the standard pipeline to query a source line using gimli:

Use goblin or direct file-io to load sections like .debug_info, .debug_line, .debug_abbrev, and .debug_str.
Wrap raw section data using gimli::EndianSlice.
Locate the Compilation Unit (CU) headers inside .debug_info.
Parse the associated line table program inside .debug_line.
Run the line table state machine to generate mapping rows containing (Address, File, Line, Column).

// Pseudo-architecture pattern for querying line info via Gimli // 1. Load DWARF payload: let mut file = File::open(path)?; let mmap = unsafe { memmap2::Mmap::map(&file)? }; let dwarf = gimli::Dwarf::load(|id| { // Read corresponding elf section matching gimli section name IDs Ok::<_, std::io::Error>(gimli::EndianSlice::new(&mmap[section_range], gimli::RunTimeEndian::Little)) })?;

// 2. Iterate through compilation units and evaluate their line programs

Knowledge Checkpoint

Define what a DIE (Debugging Information Entry) is and explain how trees of DIEs represent functions, scopes, and local variable declarations.
Describe how the DWARF Line Table state machine compacts address-to-line pairs to save binary space.
Write down the names of at least three specific debug sections required to translate an instruction pointer (RIP) address into a source line string.

Module 6: Assembling the Command-Line Debugger in Rust

In this final module, you will bring all the components together. You will build an interactive Read-Eval-Print Loop (REPL) using safe Rust abstractions, and implement debugger commands like break, continue, step, and register. You will combine the ptrace execution loop, register modifications, goblin ELF symbols, and gimli line tables into a fully functional CLI tool.

Knowledge Checkpoint

Draw the architectural boundary separating user REPL input, debug symbol engines (ELF/DWARF), and child thread state managers.
Implement the continue flow: disable the active breakpoint at the current RIP, single-step the target process to clear the original instruction, re-enable the breakpoint, and then resume execution.
Build a functioning interactive command line in Rust that successfully handles input options like register dump, break <addr>, and continue.

Course Map

Key People Index

Liz Rice (Author & open-source speaker): Known for explaining operating system containers, namespaces, and systems programming principles by building tools from scratch.
Greg Law (Co-founder of Undo): A specialist in reverse-execution technology and Linux process inspection who frequently presents on debugger mechanics.
Simon Brand (Compiler Engineer): Renowned for writing comprehensive, step-by-step guides and giving conference talks on the inner workings of compiler formats and debugger state machines.

Final Self-Assessment

Complete this checklist to verify your debugger meets all functional requirements:

Process Launching: The debugger can spawn a child target with PTRACE_TRACEME enabled and halt before execution starts.
Address-Based Breakpoints: Users can set a breakpoint at a specific hexadecimal virtual memory address.
Memory Patching: The engine successfully overwrites target instruction bytes with 0xCC and restores the original instructions when requested.
Register Access: The debugger can read and output all user-space CPU registers (including RIP, RSP, RAX) using nix::sys::ptrace::getregs.
RIP Recovery: When hitting a breakpoint, the instruction pointer is rolled back by 1 byte, allowing the original execution state to be restored.
Symbol Resolution: The debugger reads the executable's ELF symbol table via goblin and lists public functions alongside their virtual memory offsets.
Line-Number Mapping: Using the gimli crate, the debugger translates the current instruction pointer (RIP) to a file name and line number.
The REPL Loop: The interface processes user inputs like continue, step, and break without crashing or dropping process control.
Resource Cleanup: The debugger handles child exits gracefully, checking exit codes without leaving zombie processes.

Explore Further

Related Computer Science Roadmaps

View All→

computer-science

Building a Rust Debugger: ptrace, ELF & DWARF

Module 1: Systems Programming Foundations & Rust

Recommended Videos

Knowledge Checkpoint

Module 2: Process Control and the Linux Ptrace API

Recommended Videos

Gap Mitigation: Using the `nix` Crate for Safe `ptrace`

Knowledge Checkpoint

Module 3: Software Breakpoints and CPU Registers

Recommended Videos

Gap Mitigation: Register Manipulation in Rust with `nix`

Knowledge Checkpoint

Module 4: ELF Layout: Inside Linux Executables

Recommended Videos

Gap Mitigation: Programmatic ELF Parsing in Rust

Knowledge Checkpoint

Module 5: DWARF Debugging Metadata Demystified

Recommended Videos

Gap Mitigation: Using the `gimli` Crate to Parse DWARF

Knowledge Checkpoint

Module 6: Assembling the Command-Line Debugger in Rust

Recommended Videos

Knowledge Checkpoint

Course Map

Key People Index

Final Self-Assessment

Related Computer Science Roadmaps

Container Runtimes: Namespaces, Cgroups & Go

Building a CHIP-8 Emulator: Rust & CPU Basics

Building BitTorrent: Go, P2P & Concurrency

Actor Models (C++ Concurrency & Lock-Free)

Building a Rust Debugger: ptrace, ELF & DWARF

Module 1: Systems Programming Foundations & Rust

Recommended Videos

Knowledge Checkpoint

Module 2: Process Control and the Linux Ptrace API

Recommended Videos

Gap Mitigation: Using the nix Crate for Safe ptrace

Knowledge Checkpoint

Module 3: Software Breakpoints and CPU Registers

Recommended Videos

Gap Mitigation: Register Manipulation in Rust with nix

Knowledge Checkpoint

Module 4: ELF Layout: Inside Linux Executables

Recommended Videos

Gap Mitigation: Programmatic ELF Parsing in Rust

Knowledge Checkpoint

Module 5: DWARF Debugging Metadata Demystified

Recommended Videos

Gap Mitigation: Using the gimli Crate to Parse DWARF

Knowledge Checkpoint

Module 6: Assembling the Command-Line Debugger in Rust

Recommended Videos

Knowledge Checkpoint

Course Map

Key People Index

Final Self-Assessment

Related Computer Science Roadmaps

Container Runtimes: Namespaces, Cgroups & Go

Building a CHIP-8 Emulator: Rust & CPU Basics

Building BitTorrent: Go, P2P & Concurrency

Actor Models (C++ Concurrency & Lock-Free)

Gap Mitigation: Using the `nix` Crate for Safe `ptrace`

Gap Mitigation: Register Manipulation in Rust with `nix`

Gap Mitigation: Using the `gimli` Crate to Parse DWARF