QEMU & Virtualization for Kernel Dev
Master QEMU to build, boot, and debug custom Linux kernels safely — your own pocket lab where crashing the OS is just a restart away.
Overview
Every kernel developer faces the same terrifying problem on day one: the code you are writing is the operating system. There is no safety net. A bad memory write does not segfault your program — it corrupts the kernel's own data structures and takes the entire machine down with it. Hardware is expensive, reboots are slow, and nobody wants to trash their laptop just to test whether a driver probe function is correct.
The answer is virtualization. A virtual machine is a software-emulated computer that runs as an ordinary process on your host. When your experimental kernel panics inside QEMU, it looks like any other crashed process — the host keeps running, your terminal is still there, and you just restart. This transforms kernel development from a high-stakes hardware exercise into a tight, fast, iterative loop that feels more like application development.
QEMU (Quick EMUlator) is the industry-standard open-source machine emulator and virtualizer. It can simulate an entire x86_64 PC, an ARM Raspberry Pi, a RISC-V board, or dozens of other targets — all in software. Paired with KVM (Kernel-based Virtual Machine), QEMU delegates execution to the real CPU hardware for near-native speed. For kernel developers, QEMU is uniquely powerful because it exposes a GDB stub: you can freeze your entire virtual machine mid-execution and inspect every register, every page table, every kernel struct from your debugger.
This module teaches you to build a complete, minimal kernel development environment from scratch. You will compile a kernel, build a tiny root filesystem, boot them together in QEMU, and connect GDB to set breakpoints anywhere in the kernel. You will also learn virtme, a tool that makes this loop instantaneous. By the end, you will have a personal kernel lab that you can use for every module in this roadmap.
The Linux kernel codebase is 30 million lines of C. You cannot learn it by reading alone — you learn it by changing it, breaking it, and figuring out why it broke. Every technique in this roadmap (memory management, scheduling, file systems, drivers) requires you to actually run modified kernels. Without a proper virtualization workflow, each experiment might take 10-15 minutes of reboot time and risk your real data. With QEMU and the techniques in this module, a kernel change-compile-boot cycle takes under 30 seconds, and a crash is just noise.
Beyond convenience, understanding how QEMU works teaches you deep lessons about computer architecture. To make QEMU boot a kernel, you have to understand bootloaders, device trees, memory maps, and how an OS initializes hardware. The -s -S GDB workflow forces you to understand exactly where the kernel is in memory, how symbol tables work, and what "the kernel is not yet mapped" means at the instruction level. These are exactly the concepts you need to read kernel code fluently — so the tooling teaches the subject at the same time.
Kernel Dev Iteration Loop with QEMU and GDB
The complete kernel development cycle: edit source, compile, boot in QEMU, test. When a bug needs deeper analysis, attach GDB via the -s -S stub to freeze execution and inspect kernel state interactively. virtme-ng short-circuits the rootfs step for rapid daily iteration.
The concepts
01Why Develop Kernels in a VM
The kernel runs in ring 0 (x86) or EL1 (ARM) — the most privileged execution mode on the CPU. Code at this level can execute any instruction, write to any physical memory address, and reprogram hardware directly. There are no operating system protections above it, because it is the operating system.
This means bugs are catastrophic in ways application bugs are not:
- A null pointer dereference in a kernel driver does not raise SIGSEGV — it corrupts address 0 of physical memory, then the machine hangs or triple-faults.
- An off-by-one in a memory allocator corrupts the kernel's own heap, causing random crashes minutes or hours later in unrelated code.
- A forgotten spin_lock_irqrestore causes an interrupt storm that deadlocks every CPU core simultaneously.
On real hardware, every one of these requires a hard reboot, costs 30-90 seconds, and risks filesystem corruption if the kernel died mid-write. During active development you might hit 50+ crashes per day.
A virtual machine solves this by creating a fully isolated simulated computer. Your experimental kernel runs inside the VM. When it crashes, QEMU (the program simulating the VM) exits or restarts — your host kernel, terminal, editor, and filesystem are completely unaffected. The 'reboot' is just restarting a process.
The second killer feature is reproducibility. Real hardware varies — different BIOS versions, different memory timings, different PCIe enumeration orders. QEMU gives you a perfectly reproducible machine. A bug that only manifests on a specific hardware combination can be reproduced by sharing a QEMU command line.
# Proof of concept: kernel panic in QEMU vs real hardware
# On real hardware: machine locks up, requires physical power cycle
# In QEMU: the process exits, you see the panic message, you restart
# This is a tiny kernel module that panics on load (do this in QEMU ONLY)
#include <linux/init.h>
#include <linux/module.h>
static int __init panic_test_init(void)
{
panic("This is intentional — testing QEMU recovery");
return 0;
}
module_init(panic_test_init);
MODULE_LICENSE("GPL");
02QEMU Architecture: Emulation vs KVM Acceleration
QEMU has two distinct operating modes that are often confused:
Pure emulation mode (-machine pc without KVM): QEMU translates every guest instruction into host instructions at runtime using a JIT compiler called TCG (Tiny Code Generator). The guest CPU is entirely simulated in software. This is slow (10-50x slower than native) but works on any host, even if the host CPU architecture differs from the guest. You can run an ARM kernel on an x86 Mac, for example.
KVM acceleration mode (-enable-kvm on Linux, -accel hvf on macOS with Apple Silicon): When the host and guest share the same CPU architecture, QEMU uses hardware virtualization extensions (Intel VT-x / AMD-V / ARM HYP mode) to run guest code directly on the real CPU. The host kernel's KVM module sets up hardware isolation so the guest CPU thinks it has the whole machine. Guest ring-0 code runs at near-native speed. QEMU only intervenes on VM exits (I/O, traps, memory faults). This is typically 90-99% of native speed.
For kernel development on Linux x86_64, you almost always want -enable-kvm. On macOS (both Intel and Apple Silicon), use -accel hvf for hardware acceleration.
The third concept is virtio: paravirtualized devices. Real hardware emulation (e.g., an Intel e1000 NIC) requires QEMU to emulate the full hardware protocol, which is expensive. Virtio is a standardized interface where both the guest kernel driver and the QEMU backend know they are in a VM and use a shared memory ring buffer directly. Virtio block, network, and console devices are dramatically faster than full emulation. Always use them for kernel dev VMs.
Key QEMU flags you will use constantly:
-machine q35— modern PCIe-based PC machine type-cpu host— expose the real host CPU features to the guest (only with KVM)-smp 2— 2 virtual CPUs (essential for testing locking bugs)-m 512M— 512 MB RAM-kernel bzImage— directly load a Linux kernel image (bypasses bootloader)-initrd initramfs.cpio.gz— attach an initial RAM disk-append 'console=ttyS0 nokaslr'— kernel command line-nographic— no GUI window; serial console in your terminal-s -S— GDB server on port 1234, start paused (explained in depth later)
# Check if your Linux system supports KVM
ls /dev/kvm && echo "KVM available" || echo "KVM not available"
# Minimal QEMU boot with KVM (x86_64 Linux host)
qemu-system-x86_64 \
-enable-kvm \
-machine q35 \
-cpu host \
-smp 2 \
-m 512M \
-kernel arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr rdinit=/bin/sh" \
-nographic
# On macOS (Apple Silicon or Intel):
qemu-system-x86_64 \
-accel hvf \
-machine q35 \
-cpu host \
-smp 2 \
-m 512M \
-kernel arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr rdinit=/bin/sh" \
-nographic
03Compiling a Minimal Kernel for QEMU
The Linux kernel build system is Kconfig + Make. Before compiling, you must configure which features to include. The default config (defconfig) for x86_64 produces a feature-rich kernel suitable for real hardware but large and slow to compile. For kernel development, you want a minimal config that boots in QEMU quickly.
The key targets:
make defconfig— start from the architecture's default configmake tinyconfig— absolute minimum (often does not boot; needs additions)make kvmconfig(ormake kvm_guest.config) — a preset that enables virtio and KVM guest supportmake menuconfig— interactive TUI to toggle individual optionsmake localmodconfig— enable only modules currently loaded on your host (great for reducing compile time)
For kernel dev in QEMU, the recommended approach is:
- Start with
defconfig - Apply
kvmconfigfor virtio devices - Enable
CONFIG_DEBUG_INFO=yfor GDB symbols - Enable
CONFIG_GDB_SCRIPTS=yfor kernel-aware GDB helpers - Disable
CONFIG_RANDOMIZE_BASE(KASLR) to get fixed addresses for debugging
Compilation: make -j$(nproc). The output for x86_64 is arch/x86/boot/bzImage. This is a self-extracting compressed kernel image. The uncompressed kernel with debug symbols is vmlinux in the build root — you need this for GDB.
Cross-compilation: if you want to build an ARM64 kernel on an x86 host:
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j$(nproc)
The output is arch/arm64/boot/Image.
The beauty of Linux's Kconfig system is that every option is documented. Run make help and make xconfig to explore. Understanding what each option does teaches you the kernel's structure.
# Clone kernel source (use a stable release for learning)
git clone --depth=1 -b v6.6 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux
# Configure for QEMU x86_64 development
make x86_64_defconfig
make kvm_guest.config # adds virtio, KVM guest support
# Enable debug symbols and GDB support (non-interactive)
./scripts/config --enable CONFIG_DEBUG_INFO
./scripts/config --enable CONFIG_DEBUG_INFO_DWARF4
./scripts/config --enable CONFIG_GDB_SCRIPTS
./scripts/config --disable CONFIG_RANDOMIZE_BASE
./scripts/config --enable CONFIG_FRAME_POINTER
# Update config (answer 'n' to new options)
make olddefconfig
# Build (use all CPU cores)
make -j$(nproc)
# Output:
# arch/x86/boot/bzImage <- compressed boot image for QEMU -kernel
# vmlinux <- ELF with full debug symbols for GDB
ls -lh arch/x86/boot/bzImage vmlinux
04Building a Minimal Root Filesystem (initramfs)
A kernel alone cannot do anything useful — it needs a root filesystem with programs to run. When the kernel finishes initialization, it tries to execute /sbin/init (or whatever init= specifies on the command line). If there is nothing there, it panics with 'No working init found'.
For kernel development, you do not need a full Linux distro. You need something minimal that boots fast and lets you run test programs. There are three approaches:
1. initramfs with BusyBox: The fastest approach. initramfs (initial RAM filesystem) is a CPIO archive that the kernel extracts into a tmpfs at boot. BusyBox is a single binary that implements 400+ Unix utilities (sh, ls, mount, insmod, etc.) as hardlinked aliases. You can build a working root filesystem in under 1MB.
2. Buildroot: A proper cross-compilation framework that produces a complete embedded Linux system — toolchain, busybox, libraries, init system. More complex but produces a real system. Takes 15-30 minutes to build the first time.
3. Debootstrap / pre-made images: For a full Debian/Ubuntu userspace. Convenient but large and slow to boot. Better for integration testing than kernel unit testing.
For daily kernel development, initramfs + BusyBox is optimal. The workflow is:
- Build BusyBox statically linked (so it has no library dependencies)
- Create a minimal directory tree
- Write a minimal
/initshell script - Pack it into a CPIO archive and gzip it
The init script's job: mount /proc, /sys, and /dev (virtual filesystems the kernel provides), then drop to a shell. That's all you need to run kernel modules, read /proc/kallsyms, and poke kernel internals.
The reason initramfs works so well: it is entirely in RAM, so there is no disk I/O at boot. A well-built initramfs can boot to a shell in under a second in QEMU.
#!/bin/bash
# build_initramfs.sh — build a minimal initramfs with BusyBox
set -e
ROOTFS=$(mktemp -d)
# 1. Download and build BusyBox statically
wget -q https://busybox.net/downloads/busybox-1.36.1.tar.bz2
tar -xf busybox-1.36.1.tar.bz2
cd busybox-1.36.1
make defconfig
# Enable static linking
sed -i 's/# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
make oldconfig -y
make -j$(nproc)
make CONFIG_PREFIX=$ROOTFS install
cd ..
# 2. Create minimal directory structure
mkdir -p $ROOTFS/{proc,sys,dev,tmp,etc}
# 3. Write init script
cat > $ROOTFS/init << 'EOF'
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev 2>/dev/null || mdev -s
echo "Kernel: $(uname -r)"
echo "Boot successful — dropping to shell"
exec /bin/sh
EOF
chmod +x $ROOTFS/init
# 4. Pack into CPIO archive
(cd $ROOTFS && find . | cpio -oH newc | gzip -9) > /tmp/initramfs.cpio.gz
ls -lh /tmp/initramfs.cpio.gz
echo "initramfs ready"
05Booting Your Kernel in QEMU
With a bzImage and an initramfs.cpio.gz, you are ready to boot. The QEMU -kernel flag is a powerful shortcut: instead of going through a full BIOS + GRUB boot sequence, QEMU can load a Linux kernel image directly into memory and jump to it. This is called direct kernel boot and it is much faster than a real boot.
The full boot sequence in QEMU direct mode:
- QEMU loads bzImage at the correct physical address (0x100000 for x86)
- QEMU sets up a minimal E820 memory map and fake ACPI tables
- QEMU passes control to the kernel's startup code (
arch/x86/boot/header.S) - The kernel decompresses itself, sets up page tables, enables paging
start_kernel()ininit/main.cruns — this is the first C code- The kernel initializes subsystems: memory allocator, scheduler, VFS, devices
- The kernel mounts initramfs and executes
/init - Your BusyBox shell starts
The -append flag sets the kernel command line. Key parameters for development:
console=ttyS0— redirect console output to QEMU's serial port (which appears in your terminal when using-nographic)nokaslr— disable kernel address randomization (essential for GDB)rdinit=/bin/sh— run sh directly instead of /init (for ultra-minimal setups)debug— increase kernel log verbosityinitcall_debug— print every kernel initcall (fascinating for learning boot sequence)panic=5— reboot 5 seconds after a kernel panic (useful in automated testing)
Virtio networking: add -netdev user,id=n0 -device virtio-net-pci,netdev=n0 for internet access from the guest. The user backend implements NAT using SLIRP — no root privileges needed on the host.
Virtio block device: add -drive file=rootfs.img,if=virtio to attach a persistent disk. Use this when you need a full root filesystem that survives reboots.
The serial console (-nographic + console=ttyS0) is the kernel developer's best friend. You see every printk output, every panic message, every WARN_ON backtrace directly in your terminal. Exit QEMU with Ctrl-A X.
# Complete minimal boot command
qemu-system-x86_64 \
-enable-kvm \
-machine q35 \
-cpu host \
-smp 2 \
-m 512M \
-kernel linux/arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr debug" \
-nographic
# Inside the guest shell:
# uname -r → shows your kernel version
# cat /proc/cmdline → shows your append string
# dmesg | tail → last kernel messages
# ls /sys/bus → list all bus types the kernel detected
# With virtio network (NAT to host)
qemu-system-x86_64 \
-enable-kvm \
-machine q35 -cpu host -smp 2 -m 512M \
-kernel linux/arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr" \
-netdev user,id=n0 \
-device virtio-net-pci,netdev=n0 \
-nographic
# With a persistent disk image
qemu-img create -f qcow2 rootfs.qcow2 4G
qemu-system-x86_64 \
-enable-kvm \
-machine q35 -cpu host -smp 2 -m 512M \
-kernel linux/arch/x86/boot/bzImage \
-drive file=rootfs.qcow2,if=virtio,format=qcow2 \
-append "console=ttyS0 nokaslr root=/dev/vda" \
-nographic
06GDB Remote Debugging with -s -S
This is where QEMU becomes truly magical for kernel developers. The flags -s -S turn QEMU into a GDB server:
-sopens a GDB remote stub on TCP port 1234-Sstarts the virtual machine paused (frozen before executing the first instruction)
You then connect GDB from a separate terminal to the running QEMU process. GDB sends commands over the TCP connection, QEMU executes them against the virtual machine's state, and GDB receives register contents, memory values, and events.
This is exactly like a hardware JTAG debugger, but free and in software.
The critical piece: you must load vmlinux (the unstripped ELF kernel binary with DWARF debug info) into GDB, not bzImage (which is stripped and compressed). GDB reads the symbol table and DWARF debug info from vmlinux so it can map addresses to source lines.
However, there is a subtlety: when GDB first connects, the kernel's virtual address space is not yet active. Early kernel code runs in physical addresses (or identity-mapped low addresses). You must set a breakpoint at start_kernel and wait for the kernel to enable its final page tables before GDB's symbol-to-address mapping becomes reliable.
Kernel-aware GDB scripts: The kernel ships with GDB Python scripts in scripts/gdb/. These add commands like:
lx-dmesg— print the kernel ring bufferlx-ps— list all processes (likeps auxin the kernel)lx-symbols— reload symbols for all loaded kernel moduleslx-lsmod— list loaded modules
Load them with add-auto-load-safe-path /path/to/linux and source vmlinux-gdb.py.
For SMP (multi-CPU) debugging: QEMU exposes each virtual CPU as a GDB 'thread'. Use info threads to see all CPUs and thread N to switch between them. This is invaluable for debugging spinlocks and scheduler bugs.
A common workflow: set a breakpoint in a kernel function you are studying, run the kernel, hit the breakpoint, then use list, print, p *(struct task_struct *)current, x/20x $rsp to explore kernel state. The -p flag in GDB prints structures by name when debug info is loaded.
# Terminal 1: Start QEMU paused, GDB server on port 1234
qemu-system-x86_64 \
-enable-kvm \
-machine q35 -cpu host -smp 2 -m 512M \
-kernel linux/arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr" \
-nographic \
-s -S
# QEMU prints nothing and waits — it is paused at first instruction
# Terminal 2: Connect GDB
gdb linux/vmlinux
# Inside GDB:
(gdb) set architecture i386:x86-64
(gdb) target remote :1234
# GDB connects; kernel is paused at first instruction
# Load kernel GDB scripts (provides lx-dmesg, lx-ps, etc.)
(gdb) add-auto-load-safe-path /path/to/linux
(gdb) source linux/vmlinux-gdb.py
# Set breakpoint at first C function in the kernel
(gdb) break start_kernel
(gdb) continue
# Kernel boots, hits breakpoint at start_kernel
# Now you can explore
(gdb) list # show source around breakpoint
(gdb) next # step one line
(gdb) print init_task.pid # inspect init process PID (always 0)
(gdb) break do_sys_open # break on open() syscall
(gdb) continue
# After kernel is fully booted, use kernel scripts:
(gdb) lx-dmesg # print kernel log
(gdb) lx-ps # list all processes
# Inspect current process on CPU 0
(gdb) thread 1
(gdb) p $lx_current().comm # current process name
07KASLR, nokaslr, and Why Debug Addresses Must Be Fixed
KASLR (Kernel Address Space Layout Randomization) is a security feature that loads the kernel at a random physical and virtual address on every boot. This makes it much harder for an attacker who knows kernel symbol addresses to exploit a vulnerability — the addresses change every time.
For kernel development, KASLR is your enemy. Here is why:
-
GDB symbol addresses:
vmlinuxhas all symbols at their link-time addresses (e.g.,start_kernelat0xffffffff81c00000). If KASLR slides the kernel by, say,0x1200000at runtime, every symbol is now atlink_address + 0x1200000. GDB does not know this and will set breakpoints at the wrong addresses. -
printk output: Kernel messages print addresses like
RIP: 0000:ffffffff81a3b2c4. Without knowing the KASLR slide, you cannot match this to a symbol. -
/proc/kallsyms: When KASLR is enabled,
/proc/kallsymsshows 0 for all addresses unless you read it as root. Even then, matching is complex.
The fix: always add nokaslr to your kernel command line when developing. This pins the kernel at its link-time addresses. GDB, addresses in crash dumps, and /proc/kallsyms all match vmlinux.
Alternatively, if you need to test with KASLR, you can read the slide from the QEMU monitor or from /proc/kallsyms (as root) and tell GDB: add-symbol-file vmlinux <slide>. But nokaslr is simpler for day-to-day work.
A related flag: CONFIG_DEBUG_INFO_BTF=y generates BTF (BPF Type Format) type information, which is used by BPF programs and modern debugging tools like bpftrace. This is separate from DWARF debug info in vmlinux but increasingly important in the modern kernel toolchain.
# Compare: with vs without nokaslr
# Without nokaslr: every boot randomizes addresses
qemu-system-x86_64 ... -append "console=ttyS0"
# /proc/kallsyms shows randomized addresses (unpredictable)
# With nokaslr: addresses match vmlinux exactly
qemu-system-x86_64 ... -append "console=ttyS0 nokaslr"
# /proc/kallsyms output:
grep 'T start_kernel' /proc/kallsyms
# ffffffff81c00000 T start_kernel <- matches vmlinux exactly
# In GDB, verify address matches:
# (gdb) p &start_kernel
# $1 = (void (*)()) 0xffffffff81c00000 <- same address!
# If you forgot nokaslr and need to adjust GDB manually:
# 1. Find runtime address of 'linux_banner' from /proc/kallsyms
# 2. Find compile-time address: nm vmlinux | grep linux_banner
# 3. Compute slide = runtime - compile_time
# 4. In GDB: set $slide = <slide_value>
# add-symbol-file vmlinux 0xffffffff81000000+$slide
08virtme: Instant Kernel Boot Without a Root Filesystem
virtme (Virtual Machine Userspace Environment) is a Python wrapper around QEMU that solves the most tedious part of the kernel dev workflow: managing a root filesystem. Instead of building and maintaining an initramfs or disk image, virtme does something clever — it uses 9p (Plan 9 filesystem protocol) to mount your host's actual filesystem inside the guest VM.
This means:
- The guest sees your real
/home,/usr,/lib— all your host programs are available - No need to copy binaries into an initramfs
- Changes to your kernel source on the host are immediately visible in the VM
- You can run any program installed on your host system inside the VM
The workflow becomes:
- Edit kernel source
make -j$(nproc)to buildvirtme-run --kimg arch/x86/boot/bzImage --memory 512Mto boot- Test, crash, exit, goto 1
Step 3 is ~1-2 seconds. The entire cycle from edit to running kernel can be under 30 seconds.
virtme-ng (Next Generation) is the modern successor, installable via pip: pip install virtme-ng. It wraps the whole cycle into vng --build && vng — one command that builds and boots.
Limitations of virtme: because it shares the host filesystem via 9p, it is not suitable for testing filesystem-corrupting bugs (you could corrupt your real files). For that, use a disk image. Also, some kernel features that require specific hardware initialization work differently when virtme's synthetic machine is used.
When to use virtme vs raw QEMU:
- virtme: daily kernel hacking, fast iteration, syscall testing, driver development
- raw QEMU + disk image: testing boot sequence, ACPI, specific hardware emulation, filesystem code that might corrupt state
# Install virtme-ng (modern version)
pip install virtme-ng
# From inside your kernel source tree:
# Build and boot in one command:
vng --build && vng
# Or separately:
vng --build # builds the kernel
vng --memory 512M # boots current bzImage
# Pass extra kernel args:
vng -- console=ttyS0 nokaslr debug
# Mount an extra directory into the VM:
vng --overlay-dir /tmp/test-modules
# Traditional virtme (older, still widely used):
pip install virtme
virtme-run --kimg arch/x86/boot/bzImage \
--memory 512M \
--mods auto
# What virtme does internally (simplified):
# 1. Generates a tiny initramfs that sets up 9p mounts
# 2. Adds qemu args: -virtfs local,path=/,mount_tag=virtme-root,...
# 3. Boots with your bzImage
# 4. Inside VM: mount -t 9p virtme-root / -o ro,trans=virtio
# 5. Drops you into a shell with your host's /usr /home etc available
09The Fast Iteration Loop: From Edit to Running Kernel
The goal of all these tools is a tight feedback loop. Expert kernel developers iterate dozens of times per hour. Here is the complete optimized workflow:
Incremental compilation is the biggest time saver. The Linux kernel's build system tracks dependencies. If you change a single .c file, only that file and its dependents recompile. make -j$(nproc) on a changed single file typically takes 5-30 seconds, not the 10+ minutes of a full build.
ccache: Wrap your compiler with ccache. If you rebuild from a clean state (e.g., after make mrproper), ccache serves most object files from its cache. Full rebuild with warm ccache: 1-2 minutes instead of 15.
export KBUILD_BUILD_TIMESTAMP=0 # reproducible timestamps for better caching
make CC="ccache gcc" -j$(nproc)
The .config trick: Keep a known-good .config for QEMU dev. After make mrproper, copy it back with cp my_dev_config .config && make olddefconfig. Never lose your config.
Module workflow: For developing a kernel module (driver, filesystem, etc.), you often do not need to reboot the entire VM. Compile the module as a .ko file, and use insmod/rmmod to load/unload it while the VM is running. Combined with printk debugging, this is the fastest inner loop: recompile module (seconds), run rmmod + insmod, see output.
QEMU snapshots: QEMU's -snapshot flag causes all writes to go to a temporary overlay — the base image is never modified. More powerfully, the QEMU monitor (Ctrl-A + C) supports savevm and loadvm to take instant snapshots of VM state including memory. You can save a 'kernel booted, ready to test' state and restore it in milliseconds.
Serial output to host: Use -serial file:/tmp/kernel.log to log all serial output to a file. Then tail -f /tmp/kernel.log in another terminal to watch kernel messages without being inside the VM.
The full recommended daily workflow:
- Have a tmux/screen session with 3 panes: editor, build, QEMU
- Edit →
make -j$(nproc)(watch pane 2) → up-arrow to re-run QEMU (pane 3) - For modules: use a separate
make -C /path/to/linux M=$(pwd)in the module directory
# Makefile for a development kernel module (out-of-tree)
obj-m += my_driver.o
KERNEL_DIR ?= /path/to/linux
all:
make -C $(KERNEL_DIR) M=$(PWD) modules
clean:
make -C $(KERNEL_DIR) M=$(PWD) clean
load:
insmod my_driver.ko
unload:
rmmod my_driver
reload: unload load
# In the VM, after mounting 9p/virtio to access host files:
# Fast module test cycle (< 10 seconds total):
cd /path/to/my_module
make # rebuild module
make reload # unload old, load new
dmesg | tail -20 # see printk output
# QEMU snapshot workflow:
# 1. Start QEMU with a disk image (needed for savevm)
qemu-system-x86_64 -enable-kvm ... -drive file=dev.qcow2,if=virtio
# 2. Boot to shell, enter QEMU monitor
# Ctrl-A C
(qemu) savevm ready
# 3. Next session: restore instantly
(qemu) loadvm ready
# VM is back at the exact state of the shell, in milliseconds
10QEMU Machine Protocol (QMP) and the QEMU Monitor
QEMU has two interactive control interfaces beyond the VM itself:
The QEMU Monitor (Ctrl-A C in nographic mode) is an interactive console where you control the VM as if you were a hypervisor administrator. Commands:
info registers— dump all CPU registers of the current vCPUinfo mem— show current page table mappingsinfo cpus— list all vCPUsx /20x 0xffffffff81000000— examine physical memory (like GDB'sxcommand)stop/cont— pause and resume execution (like GDB's Ctrl-C andcontinue)savevm/loadvm— snapshotsquit— terminate QEMU
The monitor is useful for quick inspection without a full GDB session.
QMP (QEMU Machine Protocol) is QEMU's JSON-based remote control protocol. Enable it with -qmp unix:/tmp/qmp.sock,server,nowait. Then send JSON commands from scripts. This is how virtualization managers (libvirt, virt-manager, cloud platforms) control QEMU programmatically.
For kernel development, the most useful QMP use case is automated testing: write a script that boots QEMU, waits for a specific string in the serial output (indicating the test finished), then kills QEMU and checks the exit code. This is how kselftest and kunit tests are run in CI systems.
A minimal QMP session:
echo '{"execute":"qmp_capabilities"}' | nc -U /tmp/qmp.sock
echo '{"execute":"query-status"}' | nc -U /tmp/qmp.sock
echo '{"execute":"system_powerdown"}' | nc -U /tmp/qmp.sock
Understanding QMP is not essential for beginners but knowing it exists demystifies how cloud infrastructure (AWS, GCP) manages thousands of VMs programmatically.
# Start QEMU with QMP socket and serial output to file
qemu-system-x86_64 \
-enable-kvm -machine q35 -cpu host -smp 2 -m 512M \
-kernel linux/arch/x86/boot/bzImage \
-initrd /tmp/initramfs.cpio.gz \
-append "console=ttyS0 nokaslr" \
-serial file:/tmp/kernel.log \
-qmp unix:/tmp/qmp.sock,server,nowait \
-display none # headless, no serial in terminal
# In another terminal, watch kernel output:
tail -f /tmp/kernel.log
# Simple test automation: wait for shell prompt, then quit
python3 - << 'EOF'
import socket, json, time
# Wait for kernel to boot
for _ in range(60):
with open('/tmp/kernel.log') as f:
if '# ' in f.read(): # shell prompt
break
time.sleep(1)
# Send powerdown via QMP
sock = socket.socket(socket.AF_UNIX)
sock.connect('/tmp/qmp.sock')
sock.recv(1024) # greeting
sock.send(b'{"execute":"qmp_capabilities"}\n')
sock.recv(1024)
sock.send(b'{"execute":"system_powerdown"}\n')
print(json.loads(sock.recv(1024)))
EOF
Key insights
- ✦QEMU with -enable-kvm runs at near-native speed because guest code executes directly on the real CPU — QEMU only intercepts I/O and privilege transitions, not every instruction.
- ✦The vmlinux ELF file (with DWARF debug info) and the bzImage (compressed boot image) come from the same build but serve different purposes: bzImage boots the VM, vmlinux is for GDB. Never confuse them.
- ✦nokaslr is not optional for GDB debugging — without it, every symbol address is random at boot and your breakpoints land in wrong memory.
- ✦The kernel's initramfs is extracted into a tmpfs entirely in RAM before any disk is mounted — this is why a BusyBox initramfs can boot to a shell in under a second with no disk I/O.
- ✦virtme's 9p trick — mounting the host filesystem directly into the VM — eliminates the entire rootfs management problem for daily development and makes the edit-compile-boot cycle nearly instant.
- ✦QEMU's -s -S gives you a hardware JTAG debugger experience in software: you can freeze time, inspect every register and memory location, and single-step through kernel initialization code that runs before any OS is set up.
- ✦The kernel GDB scripts (lx-dmesg, lx-ps, lx-symbols) transform GDB from a raw memory inspector into a kernel-aware debugger that understands data structures like task_struct and linked list traversals.
- ✦For module development, you rarely need to reboot: compile the .ko on the host, insmod in the running VM, rmmod, repeat. The inner loop for driver development is measured in seconds, not minutes.
Common pitfalls
- ⚠Forgetting nokaslr: GDB connects but breakpoints never fire because they are set at compile-time addresses while the kernel loaded at a different random address. Always add nokaslr to your -append string.
- ⚠Using bzImage instead of vmlinux in GDB: bzImage is a compressed boot blob with no symbols. Always pass vmlinux to gdb and use bzImage only in the -kernel flag.
- ⚠Building without CONFIG_DEBUG_INFO=y: GDB connects but shows only raw addresses, no source lines, no struct members. You must enable debug info before building; you cannot add it to an existing binary.
- ⚠Forgetting to add console=ttyS0: the kernel boots silently with no output visible in your terminal. Always include this in -append when using -nographic.
- ⚠KVM not available on WSL or some VMs: you may get an error about /dev/kvm. Either enable nested virtualization or drop -enable-kvm (accept slower performance with TCG emulation).
- ⚠Running out of memory in the guest: small -m values cause the kernel to OOM-kill processes immediately after boot. 256M is the minimum; 512M is comfortable for most testing.
- ⚠Module .ko built against wrong kernel version: insmod will refuse to load a module built against a different kernel. Always build modules with M=$(pwd) pointing to the exact same kernel tree you booted.
- ⚠Exiting QEMU incorrectly: typing 'exit' in the guest shell only kills BusyBox, the kernel panics. To exit QEMU from -nographic mode, use Ctrl-A X.
Practice
- 01easy
Build a minimal initramfs from scratch without using the build_initramfs.sh script. Start with just a statically compiled 'hello world' C program as the /init binary. When booted in QEMU, it should print a message and then either loop or call reboot(). Document what happens if init exits without rebooting.
- 02medium
Compile the Linux kernel with CONFIG_DEBUG_INFO=y and nokaslr, boot it in QEMU with -s -S, connect GDB, and set a breakpoint in do_mkdirat() (the kernel function called by the mkdir syscall). Then in the guest shell, run 'mkdir /tmp/test' and observe GDB hit the breakpoint. Print the filename argument.
- 03medium
Write a kernel module that, when loaded, creates a /proc entry that returns the current CPU frequency in MHz. Test it by building and loading the module into a running QEMU VM without rebooting. The /proc file should be readable with 'cat /proc/cpu_freq'.
- 04hard
Set up a fully automated test: write a shell script that (1) boots a kernel in QEMU, (2) waits for a specific string in the serial output indicating boot success, (3) sends a command to the guest via the QEMU serial port, (4) reads the output, and (5) kills QEMU and exits with success or failure. This is the foundation of kernel CI testing.
- 05hard
Use QEMU's -d flag to capture an instruction-level trace of the first 1000 guest instructions during kernel startup. Analyze the trace to identify: (1) what CPU mode the kernel starts in, (2) the first memory write, and (3) when paging is enabled (the instruction that sets CR0.PG on x86). This teaches you x86 protected mode initialization.
Resources
Comprehensive reference for all QEMU flags, machine types, and device backends. The 'System Emulation' section is most relevant.
Official kernel docs explaining the vmlinux-gdb.py scripts and the full GDB workflow. Read this alongside this module.
The canonical introduction to kernel internals; no other book explains the why of kernel design as clearly.
Weekly deep-dives on kernel changes written by kernel developers for kernel developers. Invaluable for understanding context.
The modern successor to virtme. One pip install and one command away from booting your kernel with zero rootfs overhead.
Free slide decks covering kernel compilation, BusyBox, Buildroot, QEMU, and cross-compilation in exhaustive detail.
Step-by-step guide that assumes you have a working QEMU dev environment — pairs perfectly with this module.
Free textbook covering virtualization, memory management, and concurrency — the conceptual foundation under QEMU and the kernel.
Checkpoints — you can now…
- ✓You can compile a Linux kernel from source with debug info enabled and locate both bzImage and vmlinux in the build tree.
- ✓You can build a BusyBox-based initramfs from scratch, pack it as a CPIO archive, and boot it in QEMU from the command line with no GUI.
- ✓You can start QEMU with -s -S, connect GDB with the correct vmlinux, set a breakpoint at start_kernel, and step through early kernel initialization code.
- ✓You can explain why nokaslr is required for GDB debugging and what goes wrong without it.
- ✓You can use virtme-ng to boot your kernel in under 5 seconds without building or managing a root filesystem.
- ✓You can compile an out-of-tree kernel module, load it into a running QEMU VM with insmod, verify its behavior via dmesg, and unload it with rmmod — all without rebooting.
- ✓You can exit QEMU cleanly (Ctrl-A X), and you understand the difference between killing the guest init and terminating QEMU.