Kernel Debugging: Oops, Panic & Crash Analysis

An oops is the kernel's structured crash report. It is triggered when the kernel detects an unrecoverable error in kernel space — most commonly a NULL pointer dereference, a bad memory access, or an explicit BUG() call. Unlike a userspace segfault (which kills only one process), an oops in the wrong context — such as interrupt context or with a spinlock held — forces a full system panic, because there is no safe way to continue.

The oops message has a precise, well-defined structure:

1. The error description line: BUG: unable to handle kernel NULL pointer dereference at 0000000000000010. This tells you the fault type and the faulting address.
2. The IP (instruction pointer): IP: my_driver_probe+0x3c/0x80 [my_driver]. This is the most important line — it tells you the exact function, offset within that function (in hex bytes), and total function size. The [my_driver] suffix identifies a kernel module.
3. The RIP/EIP line (on x86-64): RIP: 0010:my_driver_probe+0x3c/0x80. Same information in a slightly different format.
4. Register dump: All general-purpose registers (RAX, RBX, RCX, RDX, RSI, RDI, RBP, RSP, R8–R15 on x86-64) and their values at the moment of the fault. These are indispensable for understanding what was being computed.
5. Stack trace: A Call Trace: section showing the chain of function calls that led to the fault, innermost first.
6. Modules loaded: The list of kernel modules currently loaded, which affects symbol resolution.
7. Taint flags: A string like Tainted: G W that summarizes the kernel's history (more on this below).

Here is a real minimal example of triggering an oops from a module:

// crash_demo.c — intentionally dereferences a null pointer
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>

static int __init crash_init(void)
{
    int *p = NULL;
    pr_info("About to crash...\n");
    *p = 42;  /* This dereferences NULL — triggers an oops */
    return 0;
}

static void __exit crash_exit(void) {}

module_init(crash_init);
module_exit(crash_exit);
MODULE_LICENSE("GPL");

Compile it, insmod it on a VM, and watch the oops appear in dmesg. The key insight is that the kernel does not crash immediately (unless panic_on_oops=1). By default, if the oops happens in process context (not in an interrupt or with a spinlock held), the kernel kills the offending process and continues running — giving you time to read dmesg.

The faulting address 0x10 above is not random. A NULL pointer dereference to a struct field at offset 16 (0x10) produces exactly that address. This is a diagnostic fingerprint: small addresses like 0x0 to 0x1000 almost always mean a NULL pointer dereference to some member of a NULL struct pointer.

// Reading the key fields from an oops — annotated

// BUG: unable to handle kernel NULL pointer dereference
//      at 0000000000000010
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
// Fault type + faulting virtual address (0x10 = offset 16 into a NULL struct)

// IP: my_driver_probe+0x3c/0x80 [my_driver]
//     ^^^^^^^^^^^^^^^^  ^^^^  ^^^^
//     function name     byte   total function size in bytes
//                       offset from function start

// RIP: 0010:my_driver_probe+0x3c/0x80
// CS: 0010 means ring 0 (kernel mode), 0033 would be ring 3 (user mode)

// Call Trace:
//  my_driver_probe+0x3c/0x80 [my_driver]
//  really_probe+0xd8/0x2c0
//  __driver_probe_device+0x78/0x110
//  driver_probe_device+0x1e/0x90
//  ...
//  do_init_module+0x4a/0x230
//  __do_sys_finit_module+0xa0/0x110
//  do_syscall_64+0x3d/0x90
//  entry_SYSCALL_64_after_hwframe+0x63/0xcd

printk is the kernel's printf. It is one of the oldest functions in the kernel and works even in the most hostile contexts: early boot before the memory allocator is initialized, interrupt handlers, NMI handlers, and panic paths where nothing else can be trusted. Understanding it thoroughly is essential because it is your primary debugging tool before any of the fancier infrastructure is available.

Log levels: Every printk call has a log level (priority), specified as a string prefix. There are eight levels, defined in <linux/kern_levels.h>:

The preferred modern form uses the shorthand macros: pr_err("..."), pr_warn("..."), pr_info("..."), pr_debug("..."). For driver code, device-specific variants dev_err(dev, "...") etc. automatically prefix the message with the device name.

Console log level: The file /proc/sys/kernel/printk contains four numbers: current default minimum boot-time-default. Messages with a level numerically less than (i.e., more severe than) the current console level are printed to the console in real time. Messages at or above that level go only to the ring buffer (readable via dmesg). You can set it dynamically: echo 8 > /proc/sys/kernel/printk to show all messages including KERN_DEBUG.

The ring buffer: All printk output goes to an in-kernel circular ring buffer. dmesg reads from /dev/kmsg. The buffer has a finite size (configurable via CONFIG_LOG_BUF_SHIFT, typically 256 KB to 16 MB). On a panic, the kernel attempts to flush the buffer to the console before dying.

Rate limiting: To prevent a single noisy code path from flooding the log and potentially causing a live-lock, use printk_ratelimited() or the pr_*_ratelimited() variants. By default, they allow up to 10 messages in 5 seconds.

Timestamps: Enable CONFIG_PRINTK_TIME or add printk.time=1 to the kernel command line to add timestamps to every message. This is invaluable for performance debugging.

/* printk usage patterns in a driver */
#include <linux/printk.h>

/* Old style — explicit log level prefix */
printk(KERN_ERR "my_driver: failed to allocate buffer\n");

/* Modern style — use pr_ helpers (preferred) */
pr_err("failed to allocate buffer\n");   /* KERN_ERR */
pr_warn("unexpected state %d\n", state); /* KERN_WARNING */
pr_info("probe successful\n");           /* KERN_INFO */
pr_debug("entering loop, i=%d\n", i);    /* KERN_DEBUG — compiled out
                                             unless DEBUG is defined
                                             or dynamic debug enables it */

/* Device-aware variants — prefix message with device name automatically */
dev_err(&pdev->dev, "register map init failed: %d\n", ret);
dev_info(&pdev->dev, "probed successfully\n");

/* Rate-limited — at most 10 msgs per 5 seconds */
pr_warn_ratelimited("FIFO overflow, dropping packet\n");

/* Print a hex dump of a buffer — extremely useful */
print_hex_dump_bytes("rx buf: ", DUMP_PREFIX_OFFSET, buf, len);

/* Check current console loglevel from shell:
   $ cat /proc/sys/kernel/printk
   4  4  1  7
   ^^-- current: only KERN_ERR and more severe reach console

   $ echo 8 > /proc/sys/kernel/printk   # show everything
   $ dmesg -w                           # follow new messages live */

The kernel provides a family of assertion macros that are far more powerful than a simple printk. They are the kernel equivalent of assert() in userspace, but they carry far more information and interact directly with the oops/panic machinery.

BUG(): The nuclear option. When executed, it triggers an oops unconditionally. On x86-64 it emits the ud2 (undefined instruction) opcode, which causes an invalid opcode exception, which the kernel's fault handler turns into an oops. The message includes the file, line number, and full call trace. If panic_on_oops=1, it will halt the machine. Use BUG() for truly invariant violations where continuing would corrupt data or cause security issues.

BUG_ON(condition): Equivalent to if (unlikely(condition)) BUG(). The unlikely() hint tells the compiler that the normal path is the non-BUG path, so the generated machine code keeps the fast path in line and moves the BUG handling to a cold section of code. This is important for performance: these checks appear in hot paths (allocators, schedulers, filesystem code) and must not add overhead on the fast path.

WARN() and WARN_ON(condition): A softer version. These print a full oops-style message with a stack trace, but do not kill the process or panic the kernel. Execution continues. Use WARN_ON() for conditions that should never happen but are not immediately fatal — you want to know they occurred, but the system can limp along. The output is identical in format to a BUG oops except the header says WARNING: instead of BUG:.

WARN_ON_ONCE(condition): Like WARN_ON() but only fires once per boot. This prevents log flooding if a buggy code path is hit frequently.

BUILD_BUG_ON(condition): A compile-time assertion. If condition is true, compilation fails. This is used for static invariants like structure size constraints:

BUILD_BUG_ON(sizeof(struct my_header) != 64);

WRITE_ONCE() and READ_ONCE(): Not assertions, but intimately related to correctness. They prevent the compiler from tearing or merging accesses to shared variables, which can be the subtle bug you're chasing when your WARN_ON() fires. Think of them as barriers for the compiler, not the CPU.

The pattern in the kernel is: reserve BUG_ON() for true invariants whose violation means memory is already corrupted (e.g., a reference count going negative), and use WARN_ON() for unexpected-but-recoverable states. Reviewers will push back on BUG_ON() in places where a graceful error return is possible.

/* Demonstration of BUG_ON vs WARN_ON usage patterns */

/* In a reference counting path: if this fires, the object is
   already corrupted — no point continuing */
static void obj_put(struct my_obj *obj)
{
    /* BUG_ON: refcount going negative is a use-after-free bug */
    BUG_ON(atomic_read(&obj->refcount) <= 0);
    if (atomic_dec_and_test(&obj->refcount))
        obj_free(obj);
}

/* In a lock assertion: we expect the lock to be held,
   but if it isn't we can still continue (with a warning) */
static void update_state(struct device *dev)
{
    /* WARN_ON: documents the invariant, prints trace if violated,
       but doesn't crash the machine */
    WARN_ON(!mutex_is_locked(&dev->lock));
    dev->state = STATE_READY;
}

/* WARN_ONCE: for a known-buggy-but-harmless case you want
   to be alerted about exactly once */
static irqreturn_t my_irq_handler(int irq, void *data)
{
    if (spurious_condition)
        WARN_ONCE(1, "spurious interrupt detected, irq=%d\n", irq);
    return IRQ_HANDLED;
}

/* BUILD_BUG_ON: caught at compile time, zero runtime cost */
void init_protocol(void)
{
    /* Fails compilation if struct grows beyond wire format size */
    BUILD_BUG_ON(sizeof(struct wire_header) != 32);
}

The call trace in an oops is printed as function names with hex offsets, like my_driver_probe+0x3c/0x80. To turn that into an exact source file and line number, you need the debug symbols — either a vmlinux file built with CONFIG_DEBUG_INFO=y, or a module's .ko file.

The addr2line workflow:

For kernel code (not modules), addr2line takes a virtual address from the oops and maps it back to source:

addr2line -e vmlinux -i ffffffff8123abcd

The -i flag follows inlined functions (very important — the kernel inlines aggressively, so a single address can correspond to several nested inline functions). The output gives you file.c:line_number for each frame.

For modules, the address in the oops is a relative offset, not an absolute virtual address. You need to compute the absolute address by adding the module's load base. The load base appears in the oops itself (my_driver_probe+0x3c/0x80 [my_driver]) and is also in /sys/module/my_driver/sections/.text. Then:

addr2line -e my_driver.ko 0x3c

or equivalently you can use the function name directly:

addr2line -e my_driver.ko -f -i my_driver_probe+0x3c

Using gdb for richer analysis:

gdb understands DWARF debug info and can do things addr2line cannot: display the source context, show local variable names (if not optimized away), and disassemble the exact function:

gdb vmlinux
(gdb) list *(my_driver_probe+0x3c)
(gdb) disassemble my_driver_probe
(gdb) info symbol 0xffffffff8123abcd

The list *(function+offset) form is the workhorse — it prints the source lines around the crash point.

The scripts/decode_stacktrace.sh script: The kernel ships a script at scripts/decode_stacktrace.sh that automates the addr2line process for a full oops. Feed it the oops text and point it at your vmlinux and module directory:

dmesg | ./scripts/decode_stacktrace.sh vmlinux /lib/modules/$(uname -r)/kernel

It replaces each function+offset/size [module] entry with function (file.c:line_number). This is the fastest path from raw oops to source location.

Why offsets instead of addresses? Because the kernel uses KASLR (Kernel Address Space Layout Randomization) — it loads at a random address every boot to defeat exploits. So the absolute virtual address in a running kernel is not reproducible. The offset within a function is reproducible as long as the binary hasn't changed. That is why function+0x3c/0x80 is the stable, meaningful form.

Compiler optimizations and debugging: If the kernel is built with -O2 (the default), many functions are inlined and variables are kept in registers, not on the stack. This makes addr2line give you slightly confusing output (the line it reports is often the call site of the inlined function, not the body). Building with CONFIG_DEBUG_INFO=y and ideally CONFIG_DEBUG_INFO_DWARF5=y gives the richest information. For a debugging session you can rebuild a single file with reduced optimization: make CFLAGS_my_driver.o="-Og -g" my_driver.o.

# Full workflow: from oops to source line

# 1. Capture the oops from dmesg
dmesg > /tmp/oops.txt

# 2. Identify the crashing line — look for lines like:
#    my_driver_probe+0x3c/0x80 [my_driver]

# 3. For a built-in kernel function, use addr2line on vmlinux
#    (vmlinux is the unstripped ELF, usually in the build tree)
addr2line -e /usr/lib/debug/boot/vmlinux-$(uname -r) \
          -i -f \
          ffffffff8123abcd
# Output:
# my_driver_probe
# /home/user/linux/drivers/misc/my_driver.c:47

# 4. For a module — use the .ko file directly with relative offset
addr2line -e ./my_driver.ko -i -f -j .text 0x3c
# Output:
# my_driver_probe
# /home/user/linux/drivers/misc/my_driver.c:47

# 5. Use the kernel's own decode script (fastest for full traces)
cat /tmp/oops.txt | \
  ./scripts/decode_stacktrace.sh \
  /boot/vmlinux-$(uname -r) \
  /lib/modules/$(uname -r)/kernel

# 6. gdb for interactive investigation
gdb /boot/vmlinux-$(uname -r)
# (gdb) list *(my_driver_probe+0x3c)
# (gdb) disassemble/s my_driver_probe   # /s intermixes source

Every running Linux kernel carries a taint bitmask — a set of flags that record events that might compromise the kernel's reliability or debuggability. The taint appears in every oops as a string like Tainted: G W O or Not tainted. Kernel developers pay close attention to this because a tainted kernel may be unreliable, and some taints (like loading a proprietary module) can make it impossible to rule out the taint source as the cause of the bug.

The taint flags are defined in include/linux/panic.h and each has a single-letter code. The most important ones:

You can read the current taint value:

cat /proc/sys/kernel/tainted

This returns a decimal integer (the bitmask). Zero means untainted.

You can also set taints manually for testing:

add_taint(TAINT_WARN, LOCKDEP_STILL_OK);

Why it matters for bug reporting: When you file a kernel bug report, the first thing a maintainer checks is the taint string. If P is set, they will ask you to reproduce without the proprietary module (often a GPU driver). If O is set, they will ask you to reproduce without the out-of-tree module. A clean reproduction on an untainted kernel is a prerequisite for most serious bug reports. This is not gatekeeping — it is genuinely hard to tell whether a bug is in the kernel or in a module that has tainted it.

The G flag deserves special mention: it means all loaded modules are GPL-licensed, and it is present in the taint string to actively signal the clean state. It is the absence of P.

/* Setting taint from kernel code */
#include <linux/kernel.h>
#include <linux/panic.h>  /* TAINT_* constants */

/* In a driver that does something non-standard but intentional: */
static int __init my_debug_init(void)
{
    /*
     * We're deliberately poking hardware in an unsupported way.
     * Mark the kernel so bug reporters know.
     * LOCKDEP_STILL_OK means lockdep can still trust its state.
     */
    add_taint(TAINT_USER, LOCKDEP_STILL_OK);
    pr_warn("my_debug: kernel tainted by debug hardware access\n");
    return 0;
}

/* From the shell — decode a taint value */
/* $ cat /proc/sys/kernel/tainted
   512
   That's bit 9, which is TAINT_WARN (a WARN() fired this boot)

   Full decode:
   $ awk '/Tainted/{print}' /proc/kmsg
   or look at the oops output:
   Tainted: G  W   (G=all-GPL, W=WARN fired)
*/

By default, a kernel oops in process context kills the offending task but lets the kernel continue running. This is often the right behavior for a server: one buggy process dies, but the other 10,000 keep running. However, in several important scenarios you want the kernel to panic (halt the machine) on any oops:

1. Production systems with high-availability requirements: A live kernel with an oops may be in a partially corrupted state. Rebooting (via kdump + reboot) gives you a clean slate and a memory dump to analyze.
2. Testing and CI: You want the test framework to detect the fault, not silently continue with a broken kernel.
3. Security-sensitive systems: An oops can sometimes be exploited to escalate privileges even after the immediate crash. Panicking reduces the attack surface.

Setting panic_on_oops:

# At runtime (takes effect immediately)
echo 1 > /proc/sys/kernel/panic_on_oops
# or
sysctl -w kernel.panic_on_oops=1

# Persistent (survives reboot)
echo 'kernel.panic_on_oops = 1' >> /etc/sysctl.d/99-kernel.conf

# On the kernel command line (useful for early boot crashes)
panic_on_oops=1

The panic path: When the kernel panics (whether via panic_on_oops, a direct panic() call, a double fault, or an oops in interrupt context), it executes panic() in kernel/panic.c. This function:

1. Disables preemption and interrupts
2. Prints the panic message and call trace
3. Calls registered panic notifiers (kdump hooks register here)
4. Optionally triggers a kdump (if configured)
5. Displays the panic string on the console indefinitely, or reboots after kernel.panic seconds

kernel.panic timeout: Set to a positive number to reboot that many seconds after a panic. Set to 0 to hang forever (useful for examining the console output). Set to a negative number to reboot immediately without any delay.

Panic types: Not all panics are oops-derived. Common causes include:

• Oops: general protection fault — a privileged instruction violation
• Kernel panic - not syncing: VFS: Unable to mount root fs — early boot, wrong root device
• Kernel panic - not syncing: hung_task timeout — a task held a lock for too long
• Kernel panic - not syncing: stack-protector: Kernel stack is corrupted — stack canary fired
• Kernel panic - not syncing: Fatal exception in interrupt — oops in irq context

The phrase not syncing in the panic message literally means the kernel is not attempting to sync filesystems before halting — it is not safe to do so because the system state is unknown.

/* panic() is just a function — you can call it from kernel code */
#include <linux/kernel.h>

static int critical_init(void)
{
    void __iomem *regs = ioremap(MY_HW_BASE, 0x1000);
    if (!regs)
        /*
         * Without this register map, the hardware will corrupt
         * memory. Better to halt than to continue.
         * panic() prints a message, flushes the log, and stops.
         */
        panic("critical_init: cannot map hardware registers\n");

    /* ... rest of init ... */
    return 0;
}

/* Shell: configure panic behavior */
/* Reboot 10 seconds after a panic (useful for unattended servers) */
/* $ sysctl -w kernel.panic=10 */

/* Panic on any oops (for CI or testing) */
/* $ sysctl -w kernel.panic_on_oops=1 */

/* Kernel command line equivalents (in /etc/default/grub GRUB_CMDLINE_LINUX): */
/* panic=10 panic_on_oops=1 */

The problem with pr_debug() is that it produces enormous volumes of output if left enabled. Traditionally, KERN_DEBUG messages were compiled out unless DEBUG was defined for that translation unit, meaning you had to recompile to enable them. Dynamic debug solves this beautifully: pr_debug() calls are compiled in but compiled as a two-instruction check against a per-callsite flag, which is initially zero (disabled). Enabling them costs nothing when disabled and is a single conditional branch when enabled — cheap enough to leave in production kernels.

How it works internally: Each pr_debug() call site is represented by a struct _ddebug entry compiled into a special ELF section .data.once. The kernel aggregates these into a table in /sys/kernel/debug/dynamic_debug/control. You write matching expressions to that file to enable/disable specific messages.

The control interface (/sys/kernel/debug/dynamic_debug/control):

Each line in the control file describes one pr_debug() call site:

module my_driver func my_driver_probe file drivers/misc/my_driver.c line 47 flags =_ "initializing device"

The flags field shows =_ (disabled) or =p (enabled, print). You write matching queries to the file:

# Enable all pr_debug in a module
echo 'module my_driver +p' > /sys/kernel/debug/dynamic_debug/control

# Enable a specific function
echo 'func my_driver_probe +p' > /sys/kernel/debug/dynamic_debug/control

# Enable a specific file
echo 'file drivers/misc/my_driver.c +p' > /sys/kernel/debug/dynamic_debug/control

# Enable a specific line
echo 'file drivers/misc/my_driver.c line 47 +p' > /sys/kernel/debug/dynamic_debug/control

# Disable all
echo '*:* -p' > /sys/kernel/debug/dynamic_debug/control

# Can also be set on kernel command line for early boot debugging:
# dyndbg="module my_module +p"

Additional flags beyond p (print):

• +t: add a timestamp to each message
• +m: add the module name
• +f: add the function name
• +l: add the line number
• +s: add the full source path
• +T: use thread ID instead of timestamp

Usage in code: Just use pr_debug() or dev_dbg() as normal. No special annotation is needed. The dynamic debug infrastructure automatically intercepts them if CONFIG_DYNAMIC_DEBUG=y is set.

Early boot: For debugging before the dynamic debug infrastructure is initialized, add dyndbg="module early_module +p" to the kernel command line.

The elegance of dynamic debug is that you can ship a production kernel with hundreds of pr_debug() calls in every driver, pay essentially zero cost for them normally, and then surgically enable exactly the messages you need for a specific problem — without rebooting or recompiling.

/* Using dynamic debug in driver code — nothing special required */
#include <linux/device.h>
#include <linux/module.h>

static int my_driver_probe(struct platform_device *pdev)
{
    /* This pr_debug is compiled in but costs ~nothing when disabled.
       Enable it at runtime with:
       echo 'func my_driver_probe +p' > .../dynamic_debug/control */
    pr_debug("probe called for device %s\n", dev_name(&pdev->dev));

    /* dev_dbg is the device-aware version — includes device name automatically */
    dev_dbg(&pdev->dev, "base address: %pa\n", &res->start);

    return 0;
}

/* --- Shell session enabling dynamic debug --- */

/* Mount debugfs if not already mounted */
/* # mount -t debugfs none /sys/kernel/debug */

/* View all callsites in a module */
/* # grep 'my_driver' /sys/kernel/debug/dynamic_debug/control */

/* Enable all debug messages in the module, with timestamps */
/* # echo 'module my_driver +pt' > /sys/kernel/debug/dynamic_debug/control */

/* Now watch live: */
/* # dmesg -w | grep my_driver */

/* Disable when done */
/* # echo 'module my_driver -p' > /sys/kernel/debug/dynamic_debug/control */

For production systems where you cannot attach a live debugger, kdump gives you the next best thing: a complete snapshot of the kernel's memory at the moment of the panic, saved to disk, which you can analyze offline at your leisure with the crash tool.

How kdump works: kdump exploits the kexec system call, which allows one kernel to load and boot another kernel without a hardware reset. At boot time, kdump reserves a small chunk of RAM (typically 128 MB–512 MB, controlled by the crashkernel= kernel parameter). When the running kernel panics, a crash kernel that was pre-loaded into this reserved memory is immediately booted by the kernel's panic notifier. The crash kernel sees the full memory of the panicked kernel (which is still intact — the machine did not reset), and writes it to a dump file, then reboots normally.

Setup (on a RHEL/Debian-based system):

# 1. Install kdump tools
apt install kdump-tools crash linux-image-$(uname -r)-dbg
# or on RHEL:
yum install kexec-tools crash kernel-debuginfo

# 2. Add crashkernel= to kernel command line in /etc/default/grub:
# GRUB_CMDLINE_LINUX="... crashkernel=256M"
update-grub && reboot

# 3. Verify the crash kernel is loaded:
cat /sys/kernel/kexec_crash_loaded  # should print 1

# 4. Configure dump location in /etc/kdump.conf:
path /var/crash

Triggering a test dump (non-destructive test on a VM):

# Write to sysrq trigger to force a crash
echo c > /proc/sysrq-trigger

After reboot, the dump appears in /var/crash/.

Analyzing with crash: crash is a specialized debugger for kernel core dumps. It understands kernel data structures and provides a REPL with kernel-specific commands:

crash /usr/lib/debug/boot/vmlinux-$(uname -r) \
      /var/crash/$(uname -r)/vmcore

# Inside crash:
crash> bt              # backtrace of the panicking task
crash> log            # kernel message log (dmesg at panic time)
crash> ps             # all processes at panic time
crash> vm <pid>       # virtual memory of a process
crash> struct task_struct <addr>  # decode a task_struct at address
crash> dis my_driver_probe        # disassemble a function
crash> files <pid>    # open file descriptors of a process
crash> kmem -s        # slab allocator statistics

What you can determine from a vmcore:

• The exact call trace of the panicking CPU
• The state of every other CPU at the time of the panic (crucial for deadlock analysis)
• The complete list of running processes and their states
• The contents of any kernel data structure (by address)
• Whether a lock was held and by whom
• The contents of the ring buffer at panic time

kdump + crash is the industry standard for Linux kernel post-mortem analysis. RHEL's support process requires a vmcore for most kernel bugs. Understanding how to generate and read one is a professional necessity for production systems work.

# kdump setup and crash analysis workflow

# Step 1: Verify crashkernel reservation
dmesg | grep -i 'crash'
# [    0.000000] Reserving 256MB of memory at 2048MB for crashkernel

cat /sys/kernel/kexec_crash_loaded
# 1   <-- crash kernel is loaded and ready

# Step 2: Trigger a crash on a test VM (never on production!)
echo c > /proc/sysrq-trigger
# Machine panics, crash kernel boots, writes vmcore, reboots

# Step 3: After reboot, find the dump
ls /var/crash/
# 2024-01-15-14:23:45/
#   vmcore        (the memory dump — can be several GB)
#   vmcore-dmesg.txt  (kernel log extracted automatically)

# Quick check without full crash tool:
cat /var/crash/*/vmcore-dmesg.txt | grep -A 30 'BUG\|Oops\|Panic'

# Step 4: Full analysis with crash
crash /usr/lib/debug/boot/vmlinux-$(uname -r) \
      /var/crash/*/vmcore

# Inside crash REPL:
# crash> log
# <full dmesg at time of panic>

# crash> bt
# PID: 1234  TASK: ffff888100a2b000  CPU: 2   COMMAND: "insmod"
#  #0 [ffffc90000d27d50] machine_kexec at ffffffff810648a0
#  #1 [ffffc90000d27da0] __crash_kexec at ffffffff8119e3c0
#  #2 [ffffc90000d27e70] crash_kexec at ffffffff8119e4c0
#  #3 [ffffc90000d27e88] oops_end at ffffffff8102a3d0
#  #4 [ffffc90000d27eb0] no_context at ffffffff81061e30
#  #5 [ffffc90000d27f08] __bad_area_nosemaphore at ffffffff81062070
#  #6 [ffffc90000d27f40] do_page_fault at ffffffff81062530
#  #7 [ffffc90000d27f70] page_fault at ffffffff81a01888
#    [exception RIP: crash_init+0x12]
#  #8 [ffffc90000d27fe8] crash_init at ffffffffc0000012 [crash_demo]

# crash> dis -l crash_init
# (shows disassembly with source lines if debuginfo available)

Sometimes you have an oops from a production system where you don't have access to the vmcore or a matching debuginfo package — just the raw text of the oops. Learning to read the register dump manually lets you extract significant information even in this constrained situation.

x86-64 calling convention and register roles:

• RDI: first argument
• RSI: second argument
• RDX: third argument
• RCX: fourth argument
• RAX: return value / scratch
• RBP: frame pointer (if not eliminated by -fomit-frame-pointer)
• RSP: stack pointer
• RIP: instruction pointer (where execution was at fault time)

Reading the CS register: In the oops, CS: 0010 means ring 0 (kernel mode). If you ever see CS: 0033, the fault occurred in user mode — which means the kernel was executing a syscall and the user gave it a bad address. The distinction matters a lot for diagnosis.

The CR2 register: On x86-64, when a page fault occurs, the CPU loads CR2 with the faulting virtual address. The oops prints this. When CR2 is a small number (like 0x0, 0x8, 0x10, etc.), it is almost always a NULL pointer dereference at a struct member at that byte offset. When it is a large invalid address (like 0xdeadbeefdeadbeef or 0x6b6b6b6b6b6b6b6b), it is often a use-after-free where the slab allocator has overwritten the freed memory with a poison pattern (0x6b is the SLUB poison byte).

SLUB poison values:

• 0x6b6b...: Memory that has been freed (SLUB poisons freed memory with 0x6b)
• 0x5a5a...: Memory that has been allocated but not initialized
• 0xfefefefe...: Memory that has been freed and page-freed

When you see RIP at function f+0x0/0x60, the offset is zero — meaning the crash happened at the very first instruction of the function. Combined with CR2 being a small value, this often means a NULL pointer was passed as the function's first argument (RDI is NULL), and the function immediately tried to dereference a member.

The RBX/RBP heuristic: The kernel uses RBX and R12-R15 as callee-saved registers. When you see these in the register dump, they often hold the values of local variables that the compiler chose to keep in registers. If one of them looks like a pointer into kernel memory (address in range 0xffff800000000000 and above on x86-64), you can use crash> struct <type> <address> to decode it.

Stack frame walking: Even without frame pointers (the kernel uses -fomit-frame-pointer for performance), the kernel's unwinder can walk the stack by scanning for return addresses — values that look like kernel text addresses (in the ffffffff80000000–ffffffffbfffffff range on a typical x86-64 kernel). This is how the call trace is generated even without a frame pointer chain.

# Decoding a register dump manually
# Example oops fragment:
#
# RIP: 0010:my_driver_write+0x0/0xa0 [my_driver]
# RSP: ffffc90000d27e58  EFLAGS: 00010246
# RAX: 0000000000000000 RBX: ffff888100c4a400 RCX: 0000000000000000
# RDX: 0000000000000005 RSI: 00007ffd12345678 RDI: 0000000000000000
# CR2: 0000000000000000
#
# Diagnosis:
# 1. RIP offset is +0x0 -> crashed at the very first instruction
# 2. CR2 = 0x0 -> page fault address is NULL
# 3. RDI = 0x0 -> first argument to my_driver_write() is NULL
# 4. This is: my_driver_write(NULL, user_buf, 5)
#    The function tried to dereference the NULL file pointer argument
#    (or a struct pointed to by it) at the very first instruction.
#
# RBX = 0xffff888100c4a400 -> looks like a valid kernel pointer
#   In crash: struct file 0xffff888100c4a400
#   might reveal what file struct was in play
#
# RSI = 0x00007ffd12345678 -> userspace address (bit 47 not set)
#   This is the user buffer pointer, makes sense for a write() call
#
# SLUB use-after-free example:
# RAX: 6b6b6b6b6b6b6b6b
# The 0x6b pattern means RAX holds a pointer into freed memory
# (SLUB overwrites freed slabs with 0x6b as a poison pattern)
# A subsequent dereference of RAX would crash at 0x6b6b...6b6b

# To find struct member at offset 0 (for the NULL-dereference case):
# Look at the first field of the struct that my_driver_write expects
# If the function signature is write(struct file *filp, ...)
# and filp is NULL, accessing filp->f_inode (offset 0x20 in struct file)
# would crash at CR2=0x20 — the offset tells you WHICH field was accessed

The Linux SysRq (System Request) key is a set of kernel-level commands that work even when the system is mostly unresponsive — when the normal shell is frozen, the GUI is dead, or a process is in an uninterruptible sleep. It is accessed via the keyboard combination Alt+SysRq+<letter> on a physical console, or by writing to /proc/sysrq-trigger when you have a shell.

Enabling SysRq:

echo 1 > /proc/sys/kernel/sysrq   # enable all functions
# Or a bitmask for specific functions (see Documentation/admin-guide/sysrq.rst)

Critical SysRq commands for debugging:

The safe reboot sequence: When a system is hung, the safe reboot procedure is R-E-I-S-U-B (mnemonic: Reboot Even If System Utterly Broken):

1. R: Take keyboard back from X
2. E: Terminate all processes
3. I: Kill remaining processes
4. S: Sync filesystems
5. U: Remount read-only
6. B: Reboot

Wait a few seconds between each step. This gives processes time to terminate and filesystems time to sync, minimizing the chance of filesystem corruption.

SysRq l for deadlock analysis: When a system is hung with a deadlock, Alt+SysRq+l dumps the stack trace of every CPU. This usually shows several CPUs stuck in mutex_lock or spin_lock waiting for each other — the circular dependency that defines a deadlock.

# SysRq usage from a (still-working) shell session

# Enable all SysRq functions
echo 1 > /proc/sys/kernel/sysrq

# Show backtraces of ALL CPUs — invaluable for deadlock diagnosis
echo l > /proc/sysrq-trigger
# Then read the output:
dmesg | tail -100
# Output will show:
# [12345.678] sysrq: SysRq : Show backtrace of all active CPUs
# [12345.679] NMI backtrace for cpu 0
# [12345.680] CPU: 0 PID: 1234 Comm: my_task Tainted: G W
# [12345.681] Call Trace:
# [12345.682]  __schedule+0x2d4/0x8a0
# [12345.683]  schedule+0x46/0xb0
# [12345.684]  mutex_lock+0x17/0x30
# [12345.685]  my_driver_ioctl+0x89/0x120 [my_driver]
# ...

# Show all blocked (D-state) tasks — for hung task debugging
echo w > /proc/sysrq-trigger
dmesg | grep -A 10 'sysrq.*blocked'

# Show all tasks (like 'ps' but in kernel)
echo t > /proc/sysrq-trigger

# Trigger a crash to capture vmcore (on a test VM only!)
echo c > /proc/sysrq-trigger
# ^ this WILL crash the machine

Deadlocks in the kernel are particularly insidious because they often only manifest under specific timing conditions or on specific hardware. A deadlock can lurk in production code for years, then suddenly appear on a system under unusual load. lockdep (enabled by CONFIG_PROVE_LOCKING=y) is the kernel's compile-in deadlock detector — it tracks the order in which locks are acquired and can detect potential deadlocks before they actually occur.

How it works: Every lock in the kernel (mutex, spinlock, rwlock, rcu, etc.) has a lock class (identified by the source code location where the lock variable was initialized). lockdep builds a directed graph of lock acquisition orders: if CPU 0 acquires lock A then lock B, an edge A→B is added to the graph. If CPU 1 later acquires lock B then tries to acquire lock A, lockdep detects the cycle A→B→A and reports it immediately with full stack traces for both acquisition paths — even if an actual deadlock has not occurred yet.

This is the key insight: lockdep detects potential deadlocks, not just actual ones. It is a static analysis tool running dynamically.

A lockdep warning looks like:

======================================================
WARNING: possible circular locking dependency detected
------------------------------------------------------
my_task/1234 is trying to acquire lock:
  (&my_lock_B){+.+.}, at: my_func_b+0x20/0x40

but task is already holding lock:
  (&my_lock_A){+.+.}, at: my_func_a+0x10/0x60

the existing dependency chain (in reverse order) is:
  -> (&my_lock_B){+.+.} { ... }
     -> (&my_lock_A){+.+.} { ... }

Other locking bugs lockdep catches:

• Lock-order inversions: A→B in one place, B→A in another (deadlock risk)
• Acquiring a spinlock with interrupts enabled: If an interrupt handler also takes that spinlock, a deadlock can occur if the interrupt fires while the spinlock is held
• Recursive locking: A mutex locked twice by the same task (which would deadlock)
• Locking in wrong context: A sleeping lock (mutex) acquired in atomic context (interrupt handler, spinlock held)

Performance cost: lockdep adds significant overhead (10–30% in some workloads) and is not suitable for production kernels. It is a development/debug configuration. Most distribution kernels ship with it disabled; you enable it by building your own kernel with CONFIG_PROVE_LOCKING=y.

Annotation: For complex locking patterns that lockdep cannot automatically analyze correctly (e.g., lock nesting where the outer lock controls when the inner is valid), you can annotate with lockdep_assert_held(), mutex_lock_nested(), and similar APIs. This teaches lockdep the intended nesting hierarchy.

/* A simple lock-order inversion that lockdep will catch */

static DEFINE_MUTEX(lock_A);
static DEFINE_MUTEX(lock_B);

/* Thread 1: acquires A then B */
void thread1_func(void)
{
    mutex_lock(&lock_A);
    mutex_lock(&lock_B);   /* lockdep records: A -> B dependency */
    /* ... critical section ... */
    mutex_unlock(&lock_B);
    mutex_unlock(&lock_A);
}

/* Thread 2: acquires B then A — INVERSION! */
void thread2_func(void)
{
    mutex_lock(&lock_B);
    mutex_lock(&lock_A);   /* lockdep detects B -> A, which with A -> B
                              above creates a cycle. WARN printed HERE,
                              even if no actual deadlock has happened yet */
    /* ... critical section ... */
    mutex_unlock(&lock_A);
    mutex_unlock(&lock_B);
}

/* Correct pattern: always acquire in the same global order */
void thread2_fixed(void)
{
    mutex_lock(&lock_A);   /* Same order as thread1_func */
    mutex_lock(&lock_B);
    /* ... */
    mutex_unlock(&lock_B);
    mutex_unlock(&lock_A);
}

/* Assert a lock is held — documents the invariant AND
   gives lockdep information to validate it */
void update_shared_state(void)
{
    lockdep_assert_held(&lock_A);
    /* ... safe to access lock_A-protected data ... */
}