GDB & Kernel Debugging (kgdb/KASAN)

Before wielding GDB's advanced features, you need a mental model of how it works. GDB is not magic — it uses two Linux kernel interfaces: ptrace(2) and DWARF debug information.

ptrace is a system call that lets one process (the debugger) observe and control another (the debuggee). When GDB sets a breakpoint at address 0x401234, it actually writes the single byte 0xCC (the INT3 instruction on x86) into the target process's text segment at that address. When the CPU executes 0xCC, it fires a SIGTRAP. The kernel delivers the signal to GDB (the tracer), which wakes up, restores the original byte, and waits for your continue command. This is why breakpoints in GDB have a tiny but real overhead — they patch live code in memory.

DWARF (Debugging With Attributed Record Formats) is a standard for encoding debug information into ELF binaries. When you compile with gcc -g, the compiler emits DWARF data into special ELF sections: .debug_info (types, variables, functions), .debug_line (mapping between machine addresses and source lines), .debug_frame (how to unwind the call stack at any address), and more. GDB reads these sections to answer questions like: "at address 0x401234, what source file and line am I on?" and "what is the value of local variable foo?" If you strip your binary (strip a.out), this information is gone, and GDB can still debug — but only at the assembly level.

Understanding this matters practically: optimized builds (-O2) reorder and eliminate code, so the relationship between source lines and machine code becomes approximate. A variable shown as <optimized out> means the compiler allocated it in a register that has since been reused. This is not GDB lying to you — it's the truth about what's in the binary.

# Compile with full debug info
gcc -g3 -O0 -o prog prog.c

# See the DWARF sections GDB reads
readelf --debug-dump=info prog | head -60

# See the line number table
readelf --debug-dump=line prog | head -40

# Disassemble what a breakpoint looks like:
# Before breakpoint: at 0x401150 we have the original byte e.g. 0x55 (push rbp)
# After GDB sets breakpoint: GDB writes 0xCC at 0x401150
# CPU executes 0xCC -> SIGTRAP -> GDB takes control

Most people only use GDB for break, run, next, and print. The real power starts here.

Hardware Watchpoints watch a memory address and stop execution the instant that address is read or written. Unlike software breakpoints (which only fire when the instruction pointer reaches a specific address), watchpoints fire whenever any instruction accesses the watched location. On x86-64, the CPU has four debug registers (DR0–DR3) that implement hardware watchpoints with zero runtime overhead — the CPU itself checks the condition on every memory access. This is invaluable for tracking down "who clobbered my variable?" bugs.

Conditional breakpoints avoid the tedium of hitting a breakpoint that fires thousands of times before the interesting case. break foo if x > 100 only pauses when the condition is true. GDB evaluates the condition in the debugger process, so for high-frequency code paths, a conditional break can still be slow (GDB is woken up, evaluates the condition, resumes). For those cases, prefer tracepoints or ftrace.

GDB Python scripting is transformative. GDB embeds a full Python interpreter (since GDB 7.0). You can write scripts that automate complex inspection tasks, define custom pretty-printers that display std::vector or Linux kernel list_head structures beautifully, and create custom commands. The Linux kernel source ships with scripts/gdb/ containing Python scripts that add commands like lx-ps (list all processes) and lx-dmesg (show kernel log) when debugging a kernel over QEMU.

Reverse debugging (with record and reverse-step) records every instruction the CPU executes and lets you step backwards through execution — invaluable for finding the root cause of a corruption that was written much earlier than where it was detected. The overhead is large (10x–100x slowdown), so use it for short target windows.

# --- Hardware watchpoint: stop when variable 'count' is written ---
(gdb) watch count
Hardware watchpoint 1: count
(gdb) continue
# GDB stops when 'count' changes, showing old and new value

# --- Read watchpoint ---
(gdb) rwatch arr[5]   # stop when arr[5] is READ

# --- Conditional breakpoint ---
(gdb) break process_packet if packet->type == PKT_ERROR

# --- Python pretty-printer for a simple struct ---
# In ~/.gdbinit or loaded via 'source myscript.py':
import gdb

class MyStructPrinter:
    def __init__(self, val):
        self.val = val
    def to_string(self):
        return f"MyStruct(x={self.val['x']}, y={self.val['y']})"

def lookup_type(val):
    if str(val.type) == 'struct MyStruct':
        return MyStructPrinter(val)
    return None

gdb.pretty_printers.append(lookup_type)

# --- Reverse debugging ---
(gdb) record              # start recording
(gdb) continue            # run until crash
(gdb) reverse-step        # step backward one source line
(gdb) reverse-continue    # run backward until a breakpoint

When you cannot run GDB directly on the target (embedded board, remote machine, virtual machine, or the kernel itself), you use remote debugging. The architecture is a two-process model:

1. gdbserver (or the equivalent stub) runs on the target. It is a small program that speaks the GDB Remote Serial Protocol (RSP) — a simple ASCII/binary protocol over TCP or serial. It controls the target process via ptrace and answers GDB's questions.
2. gdb runs on your host (development machine) and speaks to the stub over the network. It has the debug symbols and source code. The target only needs gdbserver — not GDB, not debug symbols, not the source tree.

The GDB Remote Serial Protocol is beautifully simple. Packets look like $<data>#<checksum>. The most common packets: ? (why did we stop?), g (read all registers), G (write all registers), m addr,len (read memory), M addr,len:data (write memory), c (continue), s (single-step), Z0,addr,len (insert breakpoint). You can literally speak this protocol by hand over a serial terminal to debug a crash on bare metal.

For embedded and kernel debugging, this same protocol is used by kgdb (the kernel's own GDB stub), JTAG debuggers (like OpenOCD), QEMU's built-in GDB server (-gdb tcp::1234), and even some microcontrollers. The workflow is always the same: target listens, host connects with target remote host:port, then you debug as if it were local.

One critical practical detail: on the host, GDB needs a binary (or vmlinux for kernel) with debug symbols that exactly matches what is running on the target. If the symbol file is even slightly different, addresses will be wrong and you will get misleading results.

# === Scenario: debug a program on a remote Linux machine ===

# On the TARGET machine (e.g. Raspberry Pi or VM):
gdbserver :2345 ./my_program arg1 arg2
# or attach to a running process:
gdbserver :2345 --attach $(pidof my_program)

# On the HOST machine (your laptop, with source + symbols):
gdb ./my_program
(gdb) target remote raspberrypi.local:2345
Remote debugging using raspberrypi.local:2345
0x00007ffff7fcd100 in ?? ()
(gdb) break main
(gdb) continue

# === QEMU built-in GDB stub (used for kernel debugging) ===
# QEMU exposes a GDB server without any software on the guest:
qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -gdb tcp::1234 \
  -S \
  -nographic ...
# -S means: freeze CPU at startup, wait for GDB connection
# -gdb tcp::1234 means: listen on port 1234

# On host:
gdb vmlinux
(gdb) target remote :1234
(gdb) break start_kernel
(gdb) continue

The Linux kernel has two built-in debugger interfaces: kgdb and kdb.

kgdb (Kernel GDB) implements the GDB Remote Serial Protocol inside the kernel. The kernel acts as its own gdbserver. When a breakpoint is hit (or a panic occurs, or you press the magic SysRq key), the kernel halts all CPUs, enters the kgdb stub, and waits for a GDB connection over a serial port or network. Your host GDB connects, loads vmlinux (the uncompressed kernel binary with full DWARF symbols), and you can:

• Set breakpoints on any kernel function (e.g., break do_sys_openat2)
• Single-step through kernel code
• Inspect kernel data structures
• Examine per-CPU variables, memory-mapped I/O, stack frames

kdb is an older, simpler interactive debugger that you access directly on the console (keyboard or serial terminal) — no second machine needed. It does not require a GDB connection. You invoke it via SysRq+g or after a panic. kdb commands: bt (backtrace), lsmod (list modules), ps (process list), md addr (memory dump), mm addr value (memory modify), go (continue). kdb is useful for quick triage when you do not have a remote GDB session set up, or on real hardware without serial access.

Setting up kgdb over QEMU is the recommended learning setup: QEMU provides a virtual serial port that your host GDB connects to. This gives you a safe, resettable environment where you can crash the kernel, reboot, and try again without touching real hardware.

Key kernel config options for kgdb/kdb:

• CONFIG_KGDB=y — enable kgdb
• CONFIG_KGDB_SERIAL_CONSOLE=y — kgdb over serial
• CONFIG_KGDB_KDB=y — enable kdb frontend
• CONFIG_DEBUG_INFO=y — DWARF symbols in vmlinux
• CONFIG_FRAME_POINTER=y — reliable stack unwinding
• CONFIG_RANDOMIZE_BASE=n — disable KASLR (optional, makes addresses predictable during learning)

A critical subtlety: when kgdb halts the kernel, it uses NMI (Non-Maskable Interrupt) to stop other CPUs. This works on x86 but can behave differently on ARM. Also, some spinlocks and interrupt handlers cannot be safely single-stepped — the kernel will deadlock if you step into a function that is holding a lock that the kgdb subsystem itself needs.

# === Kernel .config options (enable in menuconfig or .config directly) ===
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_FRAME_POINTER=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_RANDOMIZE_BASE=n   # disable KASLR for learning

# === Launch QEMU with virtual serial for kgdb ===
qemu-system-x86_64 \
  -kernel linux/arch/x86/boot/bzImage \
  -initrd rootfs.cpio.gz \
  -append "console=ttyS0 kgdboc=ttyS0,115200 kgdbwait nokaslr" \
  -serial tcp::4444,server,nowait \
  -nographic -m 512M

# kgdbwait: pause kernel at boot and wait for GDB
# kgdboc=ttyS0: use first serial port for kgdb

# === On host: connect GDB to the virtual serial port ===
gdb linux/vmlinux
(gdb) set serial baud 115200
(gdb) target remote :4444
Remote debugging using :4444
arch_kgdb_breakpoint () at arch/x86/include/asm/kgdb.h:65
(gdb) # Now you're inside the live kernel!
(gdb) break sys_openat
(gdb) continue
# In QEMU guest, run: open("hello", O_RDONLY)
# GDB will fire at the syscall entry

# === kdb: accessed directly on console ===
# Press: Alt+SysRq+g  (or echo g > /proc/sysrq-trigger)
# Entering kdb (current=..., pid 1) on processor 0 ...
kdb> bt               # backtrace current thread
kdb> ps               # list processes
kdb> lsmod            # list kernel modules
kdb> md 0xffffffff81234560  8   # dump 8 words at address
kdb> go               # continue execution

Memory corruption bugs are the most dangerous class of kernel bugs — use-after-free, buffer overflows, and out-of-bounds accesses can silently corrupt memory, leading to security vulnerabilities or unpredictable behavior hours or days after the actual bug. KASAN (Kernel Address Sanitizer) is a compile-time instrumentation technique that catches these bugs at the moment they happen.

How KASAN works: Every byte of memory that KASAN monitors has a corresponding "shadow" byte. One shadow byte encodes the validity of 8 aligned bytes of real memory (3 bits encode how many bytes in the group are valid: 0 means all 8 are valid, a negative value means inaccessible). The compiler inserts inline checks before every load and store: "before reading/writing address X, check the shadow byte for X; if it says X is invalid/freed/out-of-bounds, call kasan_report()." The kasan_report function prints a detailed report to the kernel log and typically triggers a panic (or just a warning in CONFIG_KASAN_RECOVER=y mode).

What KASAN catches:

• Use-after-free: allocating memory, freeing it, then accessing it. The shadow byte for freed memory is set to KASAN_FREE_PAGE or KASAN_KMALLOC_FREE.
• Heap buffer overflow: writing past the end of a kmalloc'd buffer. Redzones — poisoned bytes — are placed around each allocation.
• Stack buffer overflow (with KASAN stack instrumentation)
• Use-after-scope: accessing a local variable after its scope ends (GCC/Clang extension)
• Global buffer overflow

KASAN variants:

• Generic KASAN (the original): uses 1/8th of RAM for shadow memory. High overhead (~2x memory, 1.5x–3x slowdown), but catches all bugs.
• Tag-based KASAN (ARM64 only): uses hardware memory tagging (MTE on ARMv8.5). Much lower overhead. Each pointer carries a 4-bit tag; the shadow encodes the expected tag. A mismatch fires immediately.
• Software tag-based KASAN: simulates tags in software; moderate overhead.

Reading a KASAN report is a skill in itself. The report shows: the access type and size, the address, the shadow memory around it (showing which bytes are valid/freed/redzone), the stack trace of the faulting access, and the stack trace of the original allocation and free. With this information, you can immediately identify the exact object, where it was allocated, where it was freed, and where the illegal access occurred.

/* === A simple use-after-free that KASAN will catch === */
/* This is a kernel module — compile with CONFIG_KASAN=y */
#include <linux/module.h>
#include <linux/slab.h>

static int __init kasan_demo_init(void)
{
    char *buf = kmalloc(64, GFP_KERNEL);
    if (!buf)
        return -ENOMEM;

    strcpy(buf, "hello kernel");
    kfree(buf);          /* buf is now freed */

    /* BUG: use-after-free — KASAN will catch this */
    buf[0] = 'X';

    return 0;
}

module_init(kasan_demo_init);
MODULE_LICENSE("GPL");

/*
 * KASAN report (abridged) will look like:
 *
 * BUG: KASAN: use-after-free in kasan_demo_init+0x4c/0x60
 * Write of size 1 at addr ffff888101234000 by task insmod/1234
 *
 * CPU: 0 PID: 1234 Comm: insmod
 * Call Trace:
 *  kasan_report+0x...  
 *  kasan_demo_init+0x4c/0x60
 *  do_one_initcall+0x...
 *
 * Allocated by task 1234:
 *  kmalloc+0x...        <- where buf was allocated
 *  kasan_demo_init+0x...
 *
 * Freed by task 1234:
 *  kfree+0x...          <- where buf was freed
 *  kasan_demo_init+0x...
 *
 * The buggy address ffff888101234000 belongs to the object at
 * ffff888101234000 which belongs to the cache kmalloc-64
 */

A kernel oops is what happens when the kernel encounters an error but can try to recover (kill the offending process and continue). A panic is fatal — the kernel halts. Both produce a structured output to the kernel log. Learning to read this output is one of the most fundamental kernel debugging skills.

Anatomy of a kernel oops:

1. BUG: / Oops: line: the type of fault (NULL pointer dereference, general protection fault, etc.) and the instruction pointer (IP:) where it happened.
2. CPU, PID, command: which CPU core, process ID, and process name were running.
3. RIP: the instruction pointer (x86-64 calls it RIP). GDB or addr2line can map this to a source line.
4. RSP: stack pointer. Used to find local variables.
5. Register dump: all CPU registers at the time of fault.
6. Call Trace: the most valuable part. This is the kernel call stack — the chain of function calls that led to the crash. Read it bottom-up (the bottom is main/start_kernel, the top is the crashing function).
7. Code: a hex dump of the machine instructions near the crash point. The <> marks the faulting instruction.

addr2line and objdump translate raw addresses from the oops into source file + line number. addr2line -e vmlinux -i 0xffffffff810c3456 gives you fs/open.c:1234.

The decode_stacktrace.sh script (in scripts/ in the kernel tree) automates the translation: paste the raw oops text into it, and it annotates every address with its source location.

objtool and frame pointers: reliable stack unwinding requires that every function save the frame pointer on the stack (or emit DWARF CFI data). CONFIG_FRAME_POINTER=y enables this. Without it, stack traces may be incomplete or wrong. The kernel's ORC unwinder (a kernel-specific alternative to DWARF) provides fast, reliable unwinding even in interrupt context.

UBSAN (Undefined Behavior Sanitizer) is a sibling to KASAN that catches C undefined behavior at runtime: integer overflow, array-out-of-bounds, misaligned accesses. Enable with CONFIG_UBSAN=y.

# === A real-looking kernel oops (annotated) ===
#
# BUG: kernel NULL pointer dereference, address: 0000000000000010
# #PF: supervisor read access in kernel mode
# #PF: error_code(0x0000) - not-present page
# PGD 0 P4D 0
# Oops: 0000 [#1] SMP NOPTI
# CPU: 2 PID: 1337 Comm: mydriver Not tainted 6.9.0 #1
# RIP: 0010:some_driver_function+0x24/0x60   <-- crash here
# Code: 48 89 df e8 ... <48> 8b 40 10 ...   <-- <> = faulting insn
# RSP: 0018:ffffc90000d3bdc8
# RAX: 0000000000000000 RBX: ffff888100a12c00 ...
# Call Trace:
#  <TASK>
#  another_function+0x18/0x40    <-- called some_driver_function
#  device_probe+0x7c/0x120
#  really_probe+0x14c/0x3c0
#  bus_probe_device+0x95/0xa0
#  ...                           <-- read bottom-up
#  </TASK>

# === Translate an address to source line ===
addr2line -e vmlinux -i ffffffff810c3456
# Output: drivers/mydriver/mydriver.c:87

# === decode_stacktrace.sh ===
cat oops.txt | ./scripts/decode_stacktrace.sh vmlinux
# Annotates every [<addr>] with file:line

# === objdump to see the crashing instruction ===
objdump -d vmlinux | grep -A5 'some_driver_function'

Sometimes the kernel panics on a production machine and you need to analyze the crash after the fact. This is post-mortem debugging or core dump analysis. The Linux kernel can be configured to save a full memory snapshot (a vmcore file) to disk before it halts, using a mechanism called kdump.

kdump works by loading a small secondary kernel (the "capture kernel") into a reserved region of physical memory at boot. When the main kernel panics, control is transferred to the capture kernel via kexec, which runs in a pristine environment and saves the memory of the crashed kernel to /var/crash/ as a vmcore ELF file. This is the gold standard for production debugging.

The crash utility (maintained by Red Hat, open source) is a special GDB frontend designed specifically for analyzing Linux kernel vmcore files. It understands kernel data structures, per-CPU variables, memory zones, and the kernel virtual address map. You invoke it as:

crash vmlinux vmcore

crash commands:

• bt — show backtrace of the crashing task (or bt <pid> for any process)
• bt -a — backtrace of all tasks on all CPUs
• ps — list all processes at crash time
• files <pid> — open file descriptors of a process
• vm <pid> — virtual memory map of a process
• kmem -s — slab cache statistics
• struct task_struct <addr> — dump a task_struct at an address
• list <field> <addr> — walk a linked list
• rd <addr> [count] — read memory as raw words
• log — display the kernel message ring buffer at crash time
• dis <addr> — disassemble code at address

crash is invaluable for Red Hat / SUSE / enterprise systems where you cannot interactively debug with kgdb. It is also useful for analyzing QEMU VM crashes.

# === Setting up kdump on a Fedora/RHEL system ===
dnf install kexec-tools crash kernel-debuginfo

# Reserve memory for capture kernel (in /etc/default/grub):
# GRUB_CMDLINE_LINUX="... crashkernel=256M"
grub2-mkconfig -o /boot/grub2/grub.cfg

systemctl enable --now kdump

# Trigger a test crash (WARNING: crashes the machine!):
echo c > /proc/sysrq-trigger

# vmcore appears at /var/crash/<date>/vmcore

# === Analyzing the vmcore ===
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux \
      /var/crash/2026-06-08-14:23/vmcore

# crash prompts:
crash> bt
PID: 1  TASK: ffff888100010000  CPU: 0  COMMAND: "init"
 #0 [fffffe0000009e40] crash_kexec at ffffffff810ac123
 #1 [fffffe0000009eb0] oops_end at ffffffff810912ab
 ...

crash> log
# Shows dmesg at crash time

crash> ps | grep myproc
   1337  1  2  ffff888101234000  IN   0.0  myproc

crash> bt 1337
# Backtrace for pid 1337

ftrace (Function Tracer) is the kernel's own built-in tracing infrastructure, accessible through the tracefs (or debugfs) filesystem at /sys/kernel/debug/tracing/. It requires zero external tools — just shell commands and cat. It is lighter-weight than a GDB breakpoint, safe to use on production systems in many cases, and can trace millions of events per second.

ftrace works by taking advantage of the mcount/fentry instrumentation: GCC with -pg or -mfentry inserts a call to a tracer function at the start of every kernel function. When tracing is disabled, these calls are replaced with NOP instructions (5 bytes on x86) via dynamic patching — the tracing overhead is literally zero when not in use. When you enable a tracer, the kernel patches those NOPs back to calls.

Key tracers in ftrace:

• function tracer: records every kernel function call. echo function > current_tracer — produces a torrent of output, use filters.
• function_graph tracer: shows function entry AND return, with indentation showing call depth and latency. Beautiful for understanding call hierarchies.
• irqsoff tracer: measures the longest interval when interrupts were disabled (latency tracing).
• wakeup tracer: measures scheduling latency (how long from wakeup to actually running).
• trace events: structured events scattered throughout the kernel (scheduler events, block I/O events, network events, syscalls). More efficient than function tracing; use enable files under events/.

Filtering: set_ftrace_filter accepts function names with wildcards: echo 'ext4_*' > set_ftrace_filter traces only ext4 functions. set_ftrace_pid limits tracing to a specific process.

ftrace vs GDB: ftrace is non-stop — the kernel keeps running while ftrace records events. GDB stops the system at a breakpoint. For understanding flow and timing, ftrace is superior. For inspecting state (what is the value of this variable?), GDB is superior. In practice, experienced kernel developers use both: ftrace to find which code path is taken, then kgdb to set a breakpoint there and inspect state.

# All ftrace interaction is through /sys/kernel/debug/tracing/
cd /sys/kernel/debug/tracing

# === Function graph tracer ===
echo 0 > tracing_on           # disable while configuring
echo function_graph > current_tracer
echo 'do_sys_openat2' > set_graph_function  # trace this + its callees
echo 1 > tracing_on

# Perform an open() in another terminal:
cat /etc/hostname

echo 0 > tracing_on
cat trace | head -50
# Output looks like:
#  1)               |  do_sys_openat2() {
#  1)               |    getname_flags() {
#  1)   0.321 us    |      kmem_cache_alloc() ;
#  1)   0.891 us    |    }  /* getname_flags */
#  1)               |    do_filp_open() {
#  ...             
#  1)   8.234 us    |  }  /* do_sys_openat2 */

# === Trace events: watch all syscall entries ===
echo > trace                  # clear trace buffer
echo 1 > events/syscalls/sys_enter_openat/enable
echo 1 > tracing_on
cat /proc/version             # trigger an openat
echo 0 > tracing_on
cat trace
# Output:
#  bash-4321  [001] sys_enter_openat: dfd=AT_FDCWD filename=/proc/version

# === Filter by PID ===
echo $$ > set_ftrace_pid      # trace only current shell
echo function > current_tracer
echo 1 > tracing_on
ls /tmp
echo 0 > tracing_on
cat trace | head -30

# === Reset ===
echo nop > current_tracer
echo > set_ftrace_filter
echo > set_ftrace_pid

The most important practical skill in this module is building and using a complete QEMU kernel debug environment. This is the setup that Linux kernel developers use for day-to-day development and that you should use for every exercise in this module.

The setup has four components:

1. A custom-built kernel with debug options enabled (CONFIG_DEBUG_INFO, CONFIG_KGDB, CONFIG_KASAN, etc.). You build this from the kernel source tree.
2. A minimal root filesystem — a small Linux userspace that runs inside QEMU. The easiest approach is busybox static binary + cpio initramfs. You do not need a full distro.
3. QEMU running the kernel with your rootfs, exposing virtual serial ports.
4. GDB on your host, connecting to QEMU's built-in GDB stub or to kgdb via the virtual serial port.

Why QEMU and not a VM? QEMU gives you direct control over hardware: you can add/remove peripherals, simulate memory faults, pause the entire machine, snapshot and restore state, and connect GDB at any point. It is also fast to iterate: reboot a QEMU VM in under 2 seconds. Physical hardware cannot match this.

The -s -S flags are the magic: -s is shorthand for -gdb tcp::1234, and -S freezes the CPU at startup until GDB says continue. This lets you set breakpoints before the kernel has even run its first instruction.

Workflow for a debugging session:

1. Build kernel with KGDB and DEBUG_INFO
2. Launch QEMU with -s -S (frozen at boot)
3. gdb vmlinux on host, target remote :1234
4. source linux/scripts/gdb/vmlinux-gdb.py — loads kernel-aware GDB commands
5. Set breakpoints, then continue — kernel boots
6. Trigger the code path you want to debug from inside the VM
7. GDB fires, inspect state
8. Use lx-ps, lx-dmesg, lx-lsmod from the Python scripts

This environment is safe: you can crash the kernel as many times as you want. Each Ctrl+A X in QEMU kills the VM, you fix the bug, rebuild (make -j$(nproc)), and re-launch in seconds.

#!/bin/bash
# === Complete QEMU kernel debug setup script ===
# Run this from the directory containing the kernel source.

# 1. Configure the kernel with debug options
make defconfig
scripts/config \
  -e CONFIG_DEBUG_INFO \
  -e CONFIG_DEBUG_INFO_DWARF5 \
  -e CONFIG_KGDB \
  -e CONFIG_KGDB_SERIAL_CONSOLE \
  -e CONFIG_KGDB_KDB \
  -e CONFIG_KASAN \
  -e CONFIG_KASAN_GENERIC \
  -e CONFIG_FRAME_POINTER \
  -e CONFIG_MAGIC_SYSRQ \
  -d CONFIG_RANDOMIZE_BASE   # disable KASLR
make olddefconfig

# 2. Build (use all cores)
make -j$(nproc) 2>&1 | tail -5

# 3. Build a minimal busybox initramfs
# (assumes busybox is compiled statically as busybox-static)
mkdir -p initramfs/{bin,sbin,etc,proc,sys,dev}
cp busybox-static initramfs/bin/busybox
for cmd in sh ls cat mount echo; do
  ln -s busybox initramfs/bin/$cmd
done

cat > initramfs/init << 'EOF'
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
exec /bin/sh
EOF
chmod +x initramfs/init

(cd initramfs && find . | cpio -o -H newc | gzip > ../rootfs.cpio.gz)

# 4. Launch QEMU (frozen, waiting for GDB)
qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -initrd rootfs.cpio.gz \
  -append "console=ttyS0 nokaslr" \
  -nographic \
  -m 512M \
  -s -S &   # -s = gdb :1234, -S = freeze at start

QEMU_PID=$!
echo "QEMU PID: $QEMU_PID"

# 5. Connect GDB
gdb vmlinux << 'GDB_SCRIPT'
target remote :1234
add-auto-load-safe-path .
source scripts/gdb/vmlinux-gdb.py
break start_kernel
continue
GDB_SCRIPT

While not a debugger per se, KCOV (Kernel Coverage) and Syzkaller are tightly coupled to the kernel debugging ecosystem and represent the state of the art in automated kernel bug finding — responsible for thousands of CVEs.

KCOV instruments every kernel function to record which basic blocks were executed during a system call. Crucially, this information is exposed to userspace — your fuzzer can read exactly which kernel code paths were covered by each syscall. This enables coverage-guided fuzzing: the fuzzer tries random system call sequences, keeps the inputs that reach new code paths, and discards the rest. This is enormously more efficient than blind random fuzzing.

Syzkaller is Google's open-source coverage-guided kernel fuzzer. It:

1. Generates random but semantically valid system call sequences using syscall description files (.syz files) that encode the types of arguments each syscall expects.
2. Runs these sequences inside a fleet of QEMU VMs.
3. Monitors for KASAN reports, kernel panics, hangs, and other anomalies.
4. When a crash is found, it automatically minimizes the reproducer to the smallest possible program that triggers the bug.
5. Submits reproducers to syzbot, a public dashboard where kernel developers can grab and fix bugs.

For a kernel contributor, Syzkaller and syzbot are an incredible source of real bugs to fix. The syzbot.appspot.com dashboard lists hundreds of open kernel bugs, many with C reproducers attached. Fixing a syzbot bug (even a simple one) is a legitimate, high-value kernel contribution.

/* === A minimal KCOV usage example (userspace side) ===
 * Reads coverage data after a system call.
 * Requires CONFIG_KCOV=y in kernel config.
 */
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>

#define KCOV_INIT_TRACE             _IOR('c', 1, unsigned long)
#define KCOV_ENABLE                 _IO('c', 100)
#define KCOV_DISABLE                _IO('c', 101)
#define COVER_SIZE                  (64 << 10)

int main(void)
{
    int fd = open("/sys/kernel/debug/kcov", O_RDWR);
    ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE);

    uint64_t *cover = mmap(NULL, COVER_SIZE * sizeof(uint64_t),
                           PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

    ioctl(fd, KCOV_ENABLE, 0);   /* start recording */
    cover[0] = 0;                 /* reset counter */

    /* ---- syscall under test ---- */
    read(open("/proc/version", O_RDONLY), NULL, 0);
    /* ----------------------------- */

    ioctl(fd, KCOV_DISABLE, 0);  /* stop recording */

    uint64_t n = cover[0];       /* number of PCs recorded */
    printf("Covered %lu kernel PCs:\n", n);
    for (uint64_t i = 0; i < n && i < 10; i++)
        printf("  0x%llx\n", (unsigned long long)cover[i + 1]);

    /* A fuzzer would: addr2line these PCs against vmlinux,
     * keep this input if any PCs are NEW, discard otherwise */
    return 0;
}