R0KernelPath
module 098 / 262

Static Analysis & Sanitizers: sparse, smatch, KASAN, lockdep

Master the kernel's full checker arsenal — from sparse type annotations to KASAN shadow memory — so your patches arrive already proven correct by the same tools the 0day robot uses.

~18h focused study10 concepts

Overview

When you write a kernel patch, you are writing code that runs with no safety net — no memory protection, no friendly crash dialog, just a hard hang, silent data corruption, or a security hole that affects millions of machines. The Linux kernel community has therefore built an extraordinary ecosystem of automated checkers that catch bugs before they reach production: static analyzers that read your source without running it, and dynamic sanitizers that instrument the running kernel to catch bugs the moment they happen.

This module covers the full stack: sparse and smatch for static analysis at build time; Coccinelle for semantic patch transformations; and the runtime sanitizer quartet — KASAN (memory safety), KCSAN (data races), UBSAN (undefined behavior), and KMSAN (uninitialized memory) — plus lockdep (locking correctness) and kmemleak (memory leak detection). Together, these tools have found thousands of real CVEs and subsystem bugs. Every serious kernel contributor runs them routinely.

Understanding these tools is also a masterclass in the classes of bugs that matter in kernel code: use-after-free, type-confused pointer arithmetic, missing locks, TOCTOU races across the user/kernel boundary, and more. Learning to read their output will make you dramatically better at reasoning about low-level code even without the tools running.

By the end of this module you will be able to build a kernel with each tool enabled, trigger a deliberately broken driver, read the resulting reports, and submit a patch that passes the full checker suite — which is exactly what the kernel's CI system (KernelCI, 0day robot) does to every patch you send to a mailing list.

why this matters for the kernel

The kernel review process is famously rigorous. When you send a patch to lkml or a subsystem list, within hours the 0day Intel robot and KernelCI will apply it, build it with sparse/smatch/Coccinelle checks enabled, boot it on real hardware, and run the sanitizer suite. If any tool fires, your patch is rejected with a detailed report before a human even looks at it. Knowing how to silence those tools correctly (not just suppress warnings) is a hard prerequisite for getting code accepted.

More deeply, these tools encode the community's accumulated wisdom about whole classes of kernel bugs. The __user annotation in sparse exists because the kernel was repeatedly burned by missing copy_from_user() calls — direct dereference of user pointers causes either kernel crashes or exploitable privilege escalation. lockdep exists because deadlocks in interrupt context are essentially impossible to reproduce deterministically without it. KASAN exists because use-after-free bugs in the kernel are routinely turned into local privilege escalation exploits. Every annotation and every sanitizer flag is a scar from a real CVE. Learning to use them is learning that history.

Static Analysis and Sanitizer Pipeline

rendering diagram…

The two-phase checker pipeline: static analysis runs at build time (sparse, smatch, Coccinelle) and produces warnings inline with the compiler output; dynamic sanitizers run at boot time in a test VM and emit reports to dmesg. Both phases must pass before a patch is ready for submission.

The concepts

01sparse: type-safe pointer annotations

sparse is a semantic parser for C written by Linus Torvalds. Unlike a compiler, it does not generate code — it builds a full semantic model of your source and checks it for kernel-specific type violations that GCC/Clang happily ignore.

The most important mechanism is address-space annotations. In kernel C, pointers to user-space memory and pointers to kernel memory are both just void * or unsigned long at the machine level. This creates a catastrophic class of bugs: if a driver receives a user-supplied pointer and dereferences it directly (without copy_from_user()), an attacker can supply a kernel address and read/write arbitrary kernel memory — or supply a pointer to unmapped memory and crash the kernel. sparse catches this with three annotations:

  • __user — pointer lives in user virtual address space
  • __kernel — pointer lives in kernel virtual address space (the default, rarely written explicitly)
  • __iomem — pointer to memory-mapped I/O (must use readl/writel, not *ptr)

These are defined via __attribute__((noderef, address_space(N))) — the noderef part tells sparse to complain if you dereference such a pointer without first casting through the appropriate accessor. They compile to nothing; the annotation is purely for sparse.

Other important sparse checks:

  • __rcu — pointer protected by RCU; sparse (and lockdep-based CONFIG_PROVE_RCU) will warn if you read it outside of an RCU read-side critical section or without rcu_dereference().
  • __percpu — per-CPU pointer; must be accessed with this_cpu_read() or get_cpu_var().
  • __must_check / __force — __force overrides an address-space mismatch when you know it is safe (e.g., inside copy_from_user itself); __must_check annotates functions whose return values must never be ignored.
  • __bitwise — a type that forces you to be explicit about endianness. __be32 and __le32 are not the same type to sparse, so mixing them is a compile-time error. This catches entire classes of network-driver endianness bugs.

To run sparse on a kernel module: make C=1 CHECK=sparse drivers/mydriver/mydriver.o (C=1 checks only changed files; C=2 checks all files). Sparse output goes to stderr and looks like a compiler warning.

/* BAD: direct dereference of __user pointer — sparse will fire */
long bad_ioctl(unsigned long arg)
{
    int __user *uptr = (int __user *)arg;
    int val = *uptr;  /* sparse: dereference of noderef expression */
    return val;
}

/* GOOD: use copy_from_user */
long good_ioctl(unsigned long arg)
{
    int __user *uptr = (int __user *)arg;
    int val;
    if (copy_from_user(&val, uptr, sizeof(val)))
        return -EFAULT;
    return val;
}

/* RCU annotation */
struct node {
    struct rcu_head rcu;
    int data;
};
struct node __rcu *global_node;  /* __rcu annotation */

void reader(void)
{
    struct node *p;
    rcu_read_lock();
    p = rcu_dereference(global_node);  /* correct: strips __rcu, gives node* */
    if (p)
        pr_info("%d\n", p->data);
    rcu_read_unlock();
}

02smatch: interprocedural static analysis

While sparse is fast and shallow (mostly intra-function type checking), smatch does deeper interprocedural data-flow analysis. It was written by Dan Carpenter, who has used it to find tens of thousands of bugs in the kernel and is one of the most prolific kernel reviewers.

smatch builds on the sparse C parser but adds a plugin system that can track values across function calls. Its checks include:

  1. Out-of-bounds array access — smatch infers that a value derived from user input might be negative or too large before an array index operation.
  2. Kernel pointer leaks to user space — copy_to_user() of a kernel pointer, or put_user() of a stack address that contains a kernel pointer (information-leak vulnerabilities).
  3. Missing error-code checks — if a function can return a negative errno and the caller never checks, smatch flags it.
  4. Lock ordering and double-unlock — not as complete as lockdep, but catches obvious static patterns.
  5. Integer overflow before allocation — kmalloc(user_count * sizeof(struct x)) where user_count is attacker-controlled is a classic heap overflow pattern; smatch flags it.
  6. Null-pointer dereference — tracking which code paths might leave a pointer NULL.

smatch is run as part of the kernel's 0day bot and produces reports that look like: drivers/net/foo.c:123 bar() error: 'ptr' can be null. The output includes the specific code path that led to the potential null.

To run smatch locally (it requires building smatch from source and is not part of the mainline kernel tree): make CHECK="smatch -p=kernel" C=2. Dan Carpenter maintains smatch at repo.or.cz/w/smatch.git.

The practical lesson for a kernel contributor is: when smatch fires on your code, it is usually right. Follow the code path it describes, and you will almost always find a real bug or a missing bounds check.

/* smatch will flag: user-controlled size with no overflow check */
int create_buffer(unsigned int user_count)
{
    /* BUG: if user_count is large, multiplication overflows */
    void *buf = kmalloc(user_count * sizeof(long), GFP_KERNEL);
    if (!buf)
        return -ENOMEM;
    ...
}

/* FIXED: use kmalloc_array which checks for overflow */
int create_buffer_safe(unsigned int user_count)
{
    void *buf = kmalloc_array(user_count, sizeof(long), GFP_KERNEL);
    if (!buf)
        return -ENOMEM;
    ...
}

03Coccinelle: semantic patch engine

Coccinelle (French for ladybug) is a semantic patch engine developed at INRIA. Rather than just flagging bugs, it can automatically transform kernel source code. A Coccinelle script (called a SmPL script — Semantic Patch Language) describes a code pattern to match and a replacement, and Coccinelle applies it across the entire kernel tree, respecting C syntax and semantics (not just text).

This is crucial because the kernel is ~28 million lines of C. When a core API changes — say, kmalloc gains a new parameter, or a locking API is deprecated — maintainers use Coccinelle to migrate all call sites automatically, generating a single patch that touches hundreds of files consistently.

A SmPL script looks like a unified diff, but with ... (ellipsis) as a wildcard matching any code, and with identifier, expression, type metavariables. The minus (-) lines are the pattern to match; plus (+) lines are the replacement.

Beyond mechanical transformation, Coccinelle is also used for bug-finding: the kernel tree ships SmPL scripts in scripts/coccinelle/ that check for patterns like:

  • Calling kfree on a pointer that was already freed
  • Using deprecated APIs
  • Missing MODULE_LICENSE
  • Returning a non-NULL error pointer from a function declared to return ERR_PTR

To run all in-tree Coccinelle checks: make coccicheck MODE=report. To run a single script: make coccicheck COCCI=scripts/coccinelle/api/kfree_null.cocci MODE=patch. The patch mode generates an actual diff you can apply; report mode just prints findings.

Coccinelle is particularly beloved for tree-wide cleanups: you write the script once, run it, get a diff, test it, and submit. The kernel has hundreds of such automated cleanups in its history.

// scripts/coccinelle/api/devm_free.cocci (simplified illustration)
// This pattern catches devm_kzalloc paired with manual kfree
// (should use devm-managed lifecycle instead)
@@
expression dev, size, flags;
expression ptr;
@@
  ptr = devm_kzalloc(dev, size, flags);
  ...
- kfree(ptr);  // BUG: devm_kzalloc is freed automatically on device removal

// To run on the whole tree:
// make coccicheck COCCI=scripts/coccinelle/api/devm_free.cocci MODE=report

04KASAN: Kernel Address Sanitizer

KASAN (Kernel Address Sanitizer) is a dynamic memory-safety tool ported from compiler-rt's AddressSanitizer. It catches use-after-free, heap out-of-bounds, stack out-of-bounds, and global out-of-bounds bugs at the exact moment of the bad memory access, with a full stack trace. Enable it with CONFIG_KASAN=y and CONFIG_KASAN_INLINE=y.

How it works — shadow memory. KASAN dedicates 1/8 of the kernel virtual address space as a shadow map. Every 8 bytes of addressable kernel memory has a 1-byte shadow entry. The shadow byte encodes how many of those 8 bytes are currently valid (0 = all 8 valid, N = first N valid, negative = entire 8-byte region is poisoned with a specific reason).

At compile time, the compiler (GCC or Clang with -fsanitize=kernel-address) instruments every load and store with an inline shadow check:

before:  *ptr = val;
after:   shadow = addr_to_shadow(ptr);
         if (*shadow != 0) kasan_report(ptr, ...);
         *ptr = val;

This is the inline variant. The outline variant calls a function for slightly smaller code but more function-call overhead.

When memory is freed (kfree, vfree, etc.), KASAN poisons its shadow bytes with 0xfb (the "freed heap" magic). If any subsequent access hits that address, the shadow check fires, producing a report:

==================================================================
BUG: KASAN: use-after-free in mydrv_read+0x58/0x90
Read of size 4 at addr ffff888003a12340 by task cat/1234

Call Trace:
  kasan_report
  mydrv_read
  vfs_read
  ksys_read
  ...

Allocated by task 1233:
  kmalloc
  mydrv_open
  ...

Freed by task 1233:
  kfree
  mydrv_release
  ...
==================================================================

The report gives you: the bad access address and type (read/write/size), the full call stack of the bad access, the allocation stack, and the free stack. This is usually enough to diagnose the bug without a debugger.

KASAN overhead is roughly 1.5-2x memory (for shadow) and 10-50% CPU (for checks). It is only used in development/testing kernels, never in production. The canonical way to test is: boot a virtme or QEMU VM with a KASAN-enabled kernel, run your driver, look for KASAN reports in dmesg.

Generic KASAN vs. Hardware Tag-Based KASAN. On ARMv8.5+ hardware with Memory Tagging Extension (MTE), KASAN can use hardware tags instead of shadow memory, reducing overhead dramatically. This is CONFIG_KASAN_HW_TAGS.

/* Deliberately trigger a KASAN use-after-free (for testing) */
#include <linux/module.h>
#include <linux/slab.h>

static int __init kasan_test_init(void)
{
    char *buf = kmalloc(64, GFP_KERNEL);
    if (!buf)
        return -ENOMEM;
    buf[0] = 'A';  /* valid */
    kfree(buf);
    buf[0] = 'B';  /* KASAN fires here: use-after-free write */
    return 0;
}

/* Build with CONFIG_KASAN=y, then: */
/* insmod kasan_test.ko -> check dmesg for the KASAN report */

05KCSAN: Kernel Concurrency Sanitizer

KCSAN (Kernel Concurrency Sanitizer) detects data races — concurrent accesses to shared memory where at least one access is a write and there is no synchronization. Data races in C are undefined behavior, and in the kernel they cause bugs that are notoriously hard to reproduce: the race window may be nanoseconds wide on a single core but wide on a 128-core NUMA server.

KCSAN uses a watchpoint-based approach rather than shadow memory:

  1. When a memory access is instrumented (at compile time with -fsanitize=thread-style hooks), KCSAN randomly sets a watchpoint on the accessed address.
  2. If a concurrent access to the same address hits the watchpoint (on another CPU), KCSAN records the conflicting access and reports a race.
  3. The watchpoint is cleared after a short window.

This is a probabilistic approach — KCSAN does not catch every race, only races that happen to hit a watchpoint during the sampling window. But it has found hundreds of real races in the kernel.

A KCSAN report looks like:

==================================================================
BUG: KCSAN: data-race in do_something / do_something_else

write to 0xffffffff82abc120 of 4 bytes by task 123 on cpu 0:
  do_something+0x24/0x50
  ...

read to 0xffffffff82abc120 of 4 bytes by task 456 on cpu 1:
  do_something_else+0x18/0x40
  ...
==================================================================

The fix is usually adding a lock, an atomic operation, or a READ_ONCE/WRITE_ONCE if the race is benign but needs to suppress the warning (for intentional lockless algorithms documented with proper memory barriers).

When is a race "benign"? Sometimes code intentionally reads a variable without a lock as a hint — e.g., checking a flag to avoid taking a lock. The rule in the kernel is: if it is truly benign, annotate it with READ_ONCE() or data_race(), never just let it be a plain unsynchronized access. This is both for KCSAN and for the C memory model.

Enable with CONFIG_KCSAN=y. Overhead is significant (~20x in the worst case); use in dedicated test VMs.

/* Data race: two CPUs writing/reading counter without sync */
static int shared_counter;  /* BUG: no synchronization */

void increment(void) { shared_counter++; }  /* read-modify-write race */
int read_val(void)   { return shared_counter; }

/* Fix option 1: atomic */
static atomic_t safe_counter;
void increment_safe(void) { atomic_inc(&safe_counter); }

/* Fix option 2: spinlock */
static DEFINE_SPINLOCK(counter_lock);
static int locked_counter;
void increment_locked(void)
{
    spin_lock(&counter_lock);
    locked_counter++;
    spin_unlock(&counter_lock);
}

/* Fix option 3: intentionally benign hint (READ_ONCE/WRITE_ONCE) */
static int hint_flag;
void set_hint(void)  { WRITE_ONCE(hint_flag, 1); }
int get_hint(void)   { return READ_ONCE(hint_flag); }

06UBSAN: Undefined Behavior Sanitizer

UBSAN catches C undefined behavior at runtime — operations that the C standard says produce no defined result, which compilers exploit for optimization in ways that surprise programmers. In the kernel, UB usually means either a latent security bug or a subtle correctness bug that only manifests at certain optimization levels.

Kernel UBSAN (CONFIG_UBSAN=y) detects:

  • Signed integer overflow — INT_MAX + 1 is UB in C (it wraps in practice on x86, but the compiler is free to assume it never happens and elide bounds checks)
  • Shift out of bounds — 1 << 32 on a 32-bit int, or value << negative
  • Array index out of bounds (for arrays with known size)
  • Null pointer arithmetic — (char*)NULL + offset
  • Alignment violations — dereferencing a misaligned pointer (critical on architectures without hardware support for misaligned access, like some ARM configs)
  • Use of invalid enum values

A UBSAN report:

================================================================================
UBSAN: signed-integer-overflow in drivers/net/foo.c:42:18
-2147483648 - 1 cannot be represented in type 'int'
Call Trace:
  ubsan_epilogue
  handle_overflow
  foo_calc
  ...
================================================================================

UBSAN is lighter than KASAN/KCSAN — the overhead is typically 5-20% CPU with no extra memory (beyond the report buffers). It can be enabled alongside KASAN for comprehensive coverage. The most useful sub-option is CONFIG_UBSAN_SANITIZE_ALL=y to instrument all files, or per-directory with UBSAN_SANITIZE := y in a Makefile.

Why does signed overflow matter so much? A classic kernel attack pattern: user_offset + user_size can overflow to a small positive number, bypassing a bounds check that assumed the sum is large. With UBSAN, this overflow is caught immediately at test time.

/* UBSAN: signed integer overflow */
static int bad_calc(int a, int b)
{
    return a + b;  /* UBSAN fires if a=INT_MAX, b=1 */
}

/* UBSAN: shift-out-of-bounds */
static u32 bad_shift(u32 val, int n)
{
    return val << n;  /* UBSAN fires if n >= 32 or n < 0 */
}

/* Safe version: use check_add_overflow from linux/overflow.h */
#include <linux/overflow.h>
static int safe_calc(int a, int b, int *result)
{
    return check_add_overflow(a, b, result);  /* returns true if overflow */
}

07KMSAN: Kernel Memory Sanitizer

KMSAN (Kernel Memory Sanitizer) detects uses of uninitialized memory — one of the most security-relevant bug classes in the kernel, because uninitialized kernel stack or heap memory may contain secrets (passwords, keys, other processes' data) that leak to user space via copy_to_user(), network packets, or /proc files.

The canonical example: a struct is allocated with kmalloc() (which does NOT zero memory, unlike kzalloc()), some fields are filled in, and it is then copied to user space. The unfilled padding bytes between struct fields contain whatever was in that memory slot before — potentially cryptographic keys from a previous allocation. This was the root cause of many historical kernel info-leak CVEs.

KMSAN uses shadow tracking: every byte of kernel memory has a corresponding 1-bit "shadow" tracking whether it has been initialized. At compile time, every load is instrumented to check the shadow. If uninitialized data reaches copy_to_user() or is used in a branch condition, KMSAN reports it.

A KMSAN report:

BUG: KMSAN: kernel-infoleak in foo_read+0x78/0x120
  foo_read
  vfs_read
  ...
Uninitialised value was created by a heap allocation:
  kmalloc
  foo_alloc_struct
  ...

KMSAN is the newest of the four sanitizers, merged into mainline relatively recently (Clang-only, CONFIG_KMSAN=y). Its overhead is large (~3x memory, ~3x CPU) because shadow tracking for every byte is expensive. It is run primarily by automated bots (syzkaller + KMSAN) rather than interactively.

Practical rule: use kzalloc instead of kmalloc for any struct that will be copied to user space, and use memset() explicitly if you need performance. KMSAN will find the cases where you forgot.

/* BUG: uninitialized padding leaks to user space */
struct foo {
    u8  type;     /* offset 0 */
    /* 3 bytes padding -- KMSAN will flag these */
    u32 value;    /* offset 4 */
};

long bad_read(char __user *ubuf)
{
    struct foo f;
    f.type  = 1;
    f.value = 42;
    /* Padding bytes at offset 1-3 are uninitialized! */
    if (copy_to_user(ubuf, &f, sizeof(f)))  /* KMSAN fires */
        return -EFAULT;
    return sizeof(f);
}

/* FIX: zero the struct first */
long good_read(char __user *ubuf)
{
    struct foo f = {}; /* zero-initialize, including padding */
    f.type  = 1;
    f.value = 42;
    if (copy_to_user(ubuf, &f, sizeof(f)))
        return -EFAULT;
    return sizeof(f);
}

08lockdep: runtime locking validator

lockdep is the kernel's runtime locking correctness checker. It does not just catch deadlocks after they occur — it proves that your locking can never deadlock, by tracking the global lock acquisition order across all code paths and detecting violations at the first inconsistency, even if a deadlock has not yet actually occurred.

How lockdep works — lock classes and dependency graphs. lockdep assigns each type of lock (identified by the address of the spinlock_t or mutex initializer, or the DEFINE_MUTEX name) a lock class. When a lock is acquired, lockdep records:

  1. Which lock class was just acquired
  2. What other lock classes are currently held (the "held stack")

This builds a directed graph of lock ordering: an edge from A to B means "A was held when B was acquired". If lockdep ever finds a cycle in this graph (A→B→A), it immediately reports a potential deadlock, even if no thread has ever waited. The report includes the full acquisition chain that forms the cycle.

lockdep also checks:

  • IRQ safety: if lock A is ever acquired with IRQs disabled, it must always be acquired with IRQs disabled. Violating this can cause an IRQ handler to try to acquire a lock already held by the interrupted task.
  • Nested lock annotations: mutex_lock_nested() for deliberate nesting (e.g., acquiring the same lock class at different hierarchy levels, like a VFS inode lock).
  • RCU usage: CONFIG_PROVE_RCU extends lockdep to verify rcu_dereference() is called only inside RCU read-side critical sections.

A lockdep report:

============================================
WARNING: possible circular locking dependency detected
6.8.0 #1 Not tainted
--------------------------------------------
task/1234 is trying to acquire lock:
  (&B_lock){....}, at: b_func+0x20/0x40

but task/1234 already holds lock:
  (&A_lock){....}, at: a_func+0x10/0x30

which lock already depends on the new lock.
...
the existing dependency chain (in reverse):
  -> #1 (&A_lock): [acquired B then A here]
  -> #0 (&B_lock): [acquired A then B here]
============================================

lockdep is enabled by CONFIG_PROVE_LOCKING=y. It has modest overhead (~5% CPU, small memory for the dependency graph). Many kernel developers keep it on permanently in their development VMs.

An analogy: lockdep is like a building inspector who watches every door you open in a building and records the order. If you ever try to go A→B when someone else went B→A, the inspector stops you and says "this could deadlock" — even if no one has ever gotten stuck yet.

/* Classic deadlock: two threads take locks in opposite order */
static DEFINE_MUTEX(lock_A);
static DEFINE_MUTEX(lock_B);

/* Thread 1: acquires A then B */
void thread1(void)
{
    mutex_lock(&lock_A);
    mutex_lock(&lock_B);  /* lockdep records A->B dependency */
    mutex_unlock(&lock_B);
    mutex_unlock(&lock_A);
}

/* Thread 2: acquires B then A */
void thread2(void)
{
    mutex_lock(&lock_B);
    mutex_lock(&lock_A);  /* lockdep FIRES: B->A conflicts with A->B */
    mutex_unlock(&lock_A);
    mutex_unlock(&lock_B);
}

/* lockdep fires on the FIRST CALL to mutex_lock(&lock_A) inside
 * thread2 after thread1 has run, regardless of whether both
 * threads are actually running concurrently */

09kmemleak: memory leak detector

kmemleak is a runtime memory-leak detector for kernel heap allocations. Memory that is allocated with kmalloc/vmalloc but never freed creates a slow leak that can eventually exhaust kernel memory, causing system instability hours or days later — bugs that are extremely hard to debug without a tool.

kmemleak works using a conservative garbage collection approach:

  1. It maintains a tree of all outstanding kmalloc/vmalloc allocations, tracking their address and size.
  2. Periodically (or on demand), it scans all of kernel memory, CPU registers, and stack frames for values that look like pointers into allocated regions.
  3. Any allocated block that has no reachable pointer in any scanned memory is reported as a leak.

"Conservative" means it never considers something a leak if there is any memory word that happens to contain a value in the allocated range — even if that value is not actually a pointer. This means false negatives are possible, but false positives are rare.

To use kmemleak:

# Boot with CONFIG_DEBUG_KMEMLEAK=y
# Trigger your driver/subsystem
# Then:
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

A kmemleak report:

unreferenced object 0xffff888003c10000 (size 256):
  comm "insmod", pid 1234, jiffies 4294924108 (age 12.340s)
  backtrace:
    kmalloc
    mydrv_init
    do_init_module
    ...

kmemleak is also useful for auditing new drivers: write a test that repeatedly opens, exercises, and closes your device node, then scan for leaks.

One important nuance: kmemleak can be told about "false positive" pointers — if you store a pointer in an unusual way (XOR encoding, offset pointer), you can annotate it with kmemleak_not_leak() or kmemleak_ignore() to suppress spurious reports.

/* kmemleak test: allocated but pointer is overwritten without freeing */
static char *global_buf;

static int __init leak_init(void)
{
    global_buf = kmalloc(256, GFP_KERNEL);
    if (!global_buf)
        return -ENOMEM;

    /* BUG: overwrite pointer without freeing old allocation */
    global_buf = kmalloc(512, GFP_KERNEL);
    /* The first 256-byte allocation is now leaked */

    return 0;
}

/* After: echo scan > /sys/kernel/debug/kmemleak */
/* cat /sys/kernel/debug/kmemleak  -> shows the 256-byte leak */

static void __exit leak_exit(void)
{
    kfree(global_buf);  /* only frees the second allocation */
}

10Reading and acting on checker reports

Every tool in this module produces a report with a characteristic structure. Learning to read them quickly is a core skill. Here is the canonical workflow:

1. Identify the tool and bug class. Each report starts with a distinctive header:

  • BUG: KASAN: — memory safety
  • BUG: KCSAN: data-race — concurrency
  • UBSAN: — undefined behavior
  • BUG: KMSAN: — uninitialized memory
  • WARNING: possible circular locking dependency — lockdep
  • unreferenced object — kmemleak
  • sparse: ... / smatch: ... — static analysis (at build time)

2. Find the access/trigger site. The first call stack in the report is where the bad thing happened — read this first. In KASAN reports, there are usually three stacks: access, allocate, free. In lockdep, there are two chains showing the conflicting lock orderings.

3. Trace backward to the root cause. The access site is the symptom; the root cause is usually earlier — a missing lock, a premature free, a missing initialization. The allocate/free stacks in KASAN are usually more informative than the access stack.

4. Write a minimal reproducer. Before fixing, write a kernel test module (or a syzkaller snippet) that reliably triggers the report. This confirms you understand the bug.

5. Fix, then verify the tool no longer fires. Build with the same flags, run the reproducer, confirm silence.

6. Submit with Fixes: and Reported-by: tags. If the bug was found by a tool/bot: Reported-by: kernel test robot <lkp@intel.com>. If you found it yourself with a tool: note it in the commit message.

Triaging false positives. Sparse and smatch occasionally fire on correct code (e.g., when an annotation is intentionally omitted for a fast path). The correct response is not to suppress the warning — it is to add an explanatory comment and, if needed, a /* sparse:disable */ annotation with justification. Never blindly add __force or (void) casts to silence tools without understanding why they fired.

# Workflow: build and test with all checkers in a QEMU VM

# 1. Configure the kernel with all checkers
cd linux/
scripts/config -e CONFIG_KASAN -e CONFIG_KASAN_INLINE \
               -e CONFIG_KCSAN \
               -e CONFIG_UBSAN -e CONFIG_UBSAN_SANITIZE_ALL \
               -e CONFIG_DEBUG_KMEMLEAK \
               -e CONFIG_PROVE_LOCKING \
               -e CONFIG_LOCKDEP

# 2. Build with sparse
make C=2 CHECK=sparse -j$(nproc) 2>&1 | grep -E 'sparse:|error:' | head -40

# 3. Boot in QEMU (virtme is convenient)
virtme-run --kimg arch/x86/boot/bzImage --qemu-opts '-m 2G'

# 4. Inside VM: load your driver, exercise it, check dmesg
insmod mydrv.ko
dmesg | grep -E 'KASAN|KCSAN|UBSAN|KMSAN|lockdep'

# 5. Check kmemleak
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

Key insights

  • ✦Every annotation in sparse (__user, __rcu, __iomem) exists because the kernel was exploited through that exact class of pointer confusion in the past. The annotation system is a scar map of past CVEs.
  • ✦KASAN, KCSAN, UBSAN, and KMSAN are complementary — they catch different bug classes. A serious test setup runs all four simultaneously, plus lockdep. The overhead only matters for production; test VMs can afford it.
  • ✦lockdep proves deadlock-freedom proactively — it fires on the first acquisition ordering inconsistency, potentially years before an actual deadlock would ever occur on real hardware. This is one of the kernel's most impressive engineering achievements.
  • ✦The shadow byte in KASAN encoding is elegant: 0 = all 8 bytes valid, N (1-7) = only first N bytes valid, negative = poisoned with a reason code (freed heap, stack redzone, etc.). This single byte encodes both validity and diagnostic context.
  • ✦KMSAN's existence proves that struct padding is a security surface. Every struct copied to userspace must be explicitly zeroed — kzalloc is not just style preference, it is a security requirement.
  • ✦Coccinelle transforms the kernel's scale problem: when a core API changes across 28M lines of C, you write one SmPL script and generate a single consistent patch. This is why the kernel can afford to deprecate and replace APIs without leaving thousands of stale call sites.
  • ✦READ_ONCE() and WRITE_ONCE() are not just hints to KCSAN — they emit a single atomic load/store instruction and prevent the compiler from tearing or coalescing the access. They are the correct tool for intentional lockless algorithms.
  • ✦The 0day Intel robot applies sparse/smatch/Coccinelle checks to every patch sent to kernel mailing lists. Your patch will be reviewed by these tools before any human reads it. Passing them cleanly signals professionalism.

Common pitfalls

  • ⚠Adding __force or (void) casts to silence sparse without understanding why it fired. This defeats the entire purpose of the annotation system and often papers over a real security bug.
  • ⚠Confusing kzalloc (zeroed) with kmalloc (uninitialized) for structs copied to user space. KMSAN exists precisely because this mistake is extremely common and has led to dozens of info-leak CVEs.
  • ⚠Interpreting a lockdep warning as 'theoretical' and ignoring it because 'we never actually deadlocked.' lockdep warnings are real — the ordering inconsistency will eventually deadlock on a production machine under load.
  • ⚠Enabling KASAN but forgetting CONFIG_STACKTRACE or CONFIG_FRAME_POINTER — the report will be missing the allocation/free stacks, making root-cause analysis much harder.
  • ⚠Running KCSAN and dismissing every report as a 'benign race.' Most KCSAN reports are real bugs. Even if a race is truly benign, the correct fix is READ_ONCE/WRITE_ONCE, not ignoring it — the compiler may optimize away accesses it cannot see are shared.
  • ⚠Forgetting to run 'echo scan > /sys/kernel/debug/kmemleak' before reading kmemleak output. The background scan runs periodically but not instantly; you must trigger a scan after your test workload.
  • ⚠Using make C=1 (sparse) instead of C=2 when investigating new code — C=1 only checks files that have changed, so you may miss sparse warnings in files you have not recently edited.
  • ⚠Not reading past the first call stack in a KASAN report. The access stack shows the symptom; the allocate and free stacks show the cause. Fixing based only on the access site is treating the symptom.

Practice

  1. 01
    easy

    Write a minimal kernel module with a deliberate use-after-free bug: allocate a buffer in the open() file operation, free it in release(), and read from it in read(). Boot a KASAN-enabled kernel in QEMU, load your module, trigger the bug via /dev/yourdev, and paste the KASAN report. Identify and fix the bug so KASAN no longer fires.

  2. 02
    easy

    Create a struct with padding bytes (mix u8 and u32 fields), allocate it with kmalloc (not kzalloc), fill some fields, and copy it to user space with copy_to_user. Boot with CONFIG_KMSAN=y (requires Clang), trigger the copy, observe the KMSAN report. Then fix it with kzalloc or explicit memset and verify silence.

  3. 03
    medium

    Introduce a lockdep violation: two kernel threads (kthread_create) that acquire two mutexes in opposite orders (Thread A: lock_X then lock_Y; Thread B: lock_Y then lock_X). Boot with CONFIG_PROVE_LOCKING=y. You do NOT need the threads to actually run concurrently — just have both run sequentially in your init function. Observe the lockdep report and explain why lockdep fires before any deadlock actually occurs.

  4. 04
    medium

    Take any in-tree driver (pick something small like drivers/char/mem.c or a simple platform driver). Run sparse on it with 'make C=2 M=drivers/char/ 2>&1 | grep sparse'. For each sparse warning: (1) identify the address-space or annotation involved, (2) determine if it is a real bug or a false positive, (3) if real, write a patch. Submit your analysis as a short writeup.

  5. 05
    hard

    Write a Coccinelle SmPL script that finds all call sites where kmalloc() is called and the result is immediately cast to a non-void pointer type (e.g., 'p = (struct foo *)kmalloc(...)'), and transforms them to use the idiomatic no-cast form 'p = kmalloc(..., gfp)'. Run it with 'make coccicheck MODE=patch COCCI=your.cocci' on the kernel source and inspect the generated diff.

  6. 06
    hard

    Use syzkaller (or write a manual fuzzer) to exercise a simple character device driver you write. Enable KASAN + lockdep + kmemleak simultaneously. After 60 seconds of fuzzing, collect dmesg, identify any reports, triage each one (real bug vs false positive), and for each real bug: write the minimal reproducer, the root cause explanation, and the one-line fix.

Resources

📄KernelNewbies: KASAN documentation
docsfree

Official kernel documentation for KASAN — covers all variants (generic, software tag-based, hardware tag-based), configuration options, and report format.

📄Kernel documentation: Kernel debugging with sparse
docsfree

The canonical reference for sparse annotations, address-space checking, and how to run it.

📄Kernel documentation: lockdep design
docsfree

Deep technical explanation of how lockdep builds and validates the lock dependency graph — essential reading before debugging lockdep reports.

📄Kernel documentation: KCSAN
docsfree

Official KCSAN docs including the memory model, watchpoint mechanism, and how to annotate intentional races.

📄Coccinelle for Linux — official site
docsfree

Home page of the Coccinelle project with the SmPL language reference, tutorials, and links to the in-kernel scripts.

✎LWN: An introduction to lockdep
articlefree

Jonathan Corbet's accessible LWN article explaining lockdep's design — the best starting point for understanding why lockdep is so powerful.

✎LWN: The kernel address sanitizer
articlefree

Early LWN coverage of KASAN's design and rationale — good historical context for why the kernel needed its own ASan variant.

📄Kernel development tools overview
docsfree

The full index of kernel developer tools documentation, linking to KASAN, KCSAN, UBSAN, KMSAN, kmemleak, lockdep, and more.

📕Linux Device Drivers, 3rd edition (Corbet, Rubini, Kroah-Hartman)
bookfree

Freely available classic — while older, Chapters 5-8 on concurrency, locking, and memory management give the foundations you need to understand why these tools exist.

🔧syzkaller: kernel fuzzer
toolfree

The tool that has found more kernel bugs than anything else. Integrates with KASAN/KCSAN/UBSAN/KMSAN. Learning to run it locally dramatically amplifies your ability to find and reproduce bugs.

🎓Bootlin kernel debugging training materials
coursefree

Free slides + lab exercises from Bootlin's kernel debugging course, covering KASAN, lockdep, kmemleak, and ftrace in a hands-on format.

📄KMSAN kernel documentation
docsfree

Official KMSAN docs explaining the shadow tracking mechanism, Clang dependency, and how to use it to catch uninitialized-memory info-leaks.

Checkpoints — you can now…

  • ✓You can build a kernel with CONFIG_KASAN=y, CONFIG_PROVE_LOCKING=y, and CONFIG_UBSAN=y simultaneously and explain the trade-offs of each.
  • ✓You can read a KASAN use-after-free report and immediately identify the allocation site, free site, and access site from the three call stacks.
  • ✓You can run 'make C=2 CHECK=sparse' on a kernel module and correctly triage each sparse warning as a real bug, a false positive, or a documentation gap.
  • ✓You can write a Coccinelle SmPL script that finds and transforms a simple code pattern across the kernel tree.
  • ✓You can explain why lockdep fires before a deadlock actually occurs, and trace a lockdep circular-dependency report to the two conflicting acquisition orderings.
  • ✓You can identify from a KMSAN report which struct field or padding byte caused an information leak, and apply the minimal fix.
  • ✓You can trigger a kmemleak scan on demand and interpret an 'unreferenced object' report to find the allocation that was leaked.
  • ✓You can explain to someone else why READ_ONCE() and WRITE_ONCE() are both a KCSAN annotation and a compiler/architecture correctness requirement.