Kernel Testing: kselftests, KUnit & syzkaller

In userspace, testing is relatively easy: spawn a process, call a function, assert the return value, catch exceptions on failure. The kernel cannot do any of that. There is no safety net below ring 0. An assertion failure cannot throw an exception — it must either BUG() (panic the machine) or WARN() (print a trace and continue). Memory allocation can fail or be forbidden (inside a spinlock-held section, for instance). There is no standard library. The kernel runs in a shared address space where a bad pointer write corrupts everyone's state simultaneously.

This forces a fundamentally different testing philosophy:

1. Tests must be conservative about what they assert. A test that panics the kernel on failure is only useful if you are willing to reboot.
2. Tests must respect execution context. You cannot call kmalloc(GFP_KERNEL) inside an interrupt handler. Tests must be aware of whether they run in process context, atomic context, or a dedicated kernel thread.
3. Isolation is hard. Global kernel state (e.g., the VFS mount table, the network stack, slab caches) is shared between tests. KUnit goes to lengths to allow test suites to set up and tear down fixtures, but full isolation requires careful design.
4. Output is constrained. Test results must flow through printk, which is rate-limited, asynchronous, and not guaranteed to reach userspace in order.

The kernel's answer to these challenges is layered: kselftest runs mostly in userspace and probes kernel behavior through the system call interface (much safer); KUnit runs inside the kernel but in a controlled, carefully scoped context; syzkaller probes the kernel from outside with adversarial inputs.

Understanding this layering is the first conceptual step. Each tool occupies a different position in the risk/depth tradeoff.

kselftest (found in tools/testing/selftests/ in the kernel source tree) is the kernel's built-in integration test suite. Each test is a userspace program (usually C, sometimes shell) that exercises a kernel interface — a system call, a /proc entry, a sysfs file, a socket option — and checks that the behavior matches the specification.

The key insight: kselftest tests do not run in the kernel. They run as normal userspace processes and interact with the kernel through its published ABIs. This makes them safe (a test crash is just a process crash, not a kernel panic), portable (they can run on any kernel that supports the interface), and easy to write (you can use glibc, pthreads, standard I/O, etc.).

Structure of a kselftest: Each subdirectory under tools/testing/selftests/ corresponds to a kernel subsystem (e.g., mm for memory management, net for networking, futex for futexes). Each test directory contains a Makefile and one or more test programs. Tests report results using kselftest.h helpers that emit TAP (Test Anything Protocol) lines to stdout.

Running kselftests:

# From kernel source root, build and run a single subsystem:
make -C tools/testing/selftests/mm run_tests

# Run all selftests (slow, requires root for some):
make -C tools/testing/selftests run_tests

# Run in a VM using the kernel's built-in runner:
./tools/testing/selftests/run_kselftest.sh

TAP output: kselftest uses the TAP13 format. A passing test prints ok 1 test_name, a failing one prints not ok 1 test_name. The framework counts tests and produces a summary. KTAP (Kernel TAP) is the kernel's extension of TAP for nested test suites, which we cover later.

kselftest is also the home of regression tests for kernel ABI. If a kernel change breaks a syscall's documented behavior, there is probably already a kselftest for it. This makes kselftest the first line of defense against ABI regressions.

/* tools/testing/selftests/example/test_basic.c
 * A minimal kselftest showing the idiom.
 * Build: gcc -o test_basic test_basic.c -I../../.. -lkselftest
 * (In practice the Makefile handles this)
 */
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include "kselftest.h"

static void test_getpid_returns_positive(void)
{
    pid_t pid = getpid();
    if (pid > 0)
        ksft_test_result_pass("getpid returns positive value: %d\n", pid);
    else
        ksft_test_result_fail("getpid returned %d\n", pid);
}

static void test_getuid_consistent(void)
{
    uid_t uid1 = getuid();
    uid_t uid2 = getuid();
    if (uid1 == uid2)
        ksft_test_result_pass("getuid is idempotent\n");
    else
        ksft_test_result_fail("getuid returned different values: %u vs %u\n",
                              uid1, uid2);
}

int main(void)
{
    ksft_print_header();
    ksft_set_plan(2);  /* announce: 2 tests total */

    test_getpid_returns_positive();
    test_getuid_consistent();

    ksft_finished();   /* exit with correct TAP exit code */
}

KUnit (merged in Linux 5.5, 2019) is a unit testing framework that runs inside the kernel. Unlike kselftest, KUnit tests are kernel modules (or built into the kernel image) written in C using a JUnit-style API. They test internal kernel functions directly — things that are never exported to userspace and have no system call interface.

Why test inside the kernel? Because many kernel bugs live in internal helper functions: bit manipulation utilities, data structure operations, parser logic, arithmetic on strange types. You cannot test idr_alloc() from userspace — it is a private kernel API. KUnit lets you call these functions directly in a controlled test harness.

KUnit execution model: KUnit tests run in a specially-managed kernel thread context. The framework provides:

• KUNIT_EXPECT_* macros: non-fatal assertions. On failure, the test is marked failed and execution continues.
• KUNIT_ASSERT_* macros: fatal assertions. On failure, the current test case aborts (but other test cases still run).
• Fixtures: struct kunit_case arrays group tests into suites. Each suite can have init and exit hooks for setup/teardown.
• Resource management: kunit_kmalloc automatically frees allocations at test end, preventing leaks even when a test aborts.

Running KUnit tests: The easiest way to develop KUnit tests is the kunit_tool (tools/testing/kunit/kunit.py), which builds and runs a UML (User Mode Linux) kernel — the kernel running as a userspace process. This means no VM needed, near-instant rebuild cycles, and full debugger support.

# Run all KUnit tests using UML:
./tools/testing/kunit/kunit.py run

# Run only tests matching a glob pattern:
./tools/testing/kunit/kunit.py run 'list*'

# Run on real hardware (results appear in dmesg):
cat /sys/kernel/debug/kunit/results   # after booting with CONFIG_KUNIT=y

Where do KUnit tests live? KUnit tests live alongside the code they test. A driver in drivers/foo/bar.c has its KUnit tests in drivers/foo/bar_test.c. This proximity is intentional — it makes tests easier to find and update when the code changes.

KTAP output: KUnit produces KTAP (Kernel TAP) output, which extends TAP with nesting support. A suite is a top-level TAP plan, and each test case is an entry within it. This allows test runners to aggregate results hierarchically.

// lib/math/rational_kunit_test.c  (simplified from real kernel)
// Tests the rational number reduction code in lib/math/rational.c

#include <kunit/test.h>
#include <linux/rational.h>

static void rational_test_basic(struct kunit *test)
{
    unsigned long num, den;

    /* 2/4 should reduce to 1/2 */
    rational_best_approximation(2, 4, ULONG_MAX, ULONG_MAX, &num, &den);
    KUNIT_EXPECT_EQ(test, num, 1UL);
    KUNIT_EXPECT_EQ(test, den, 2UL);
}

static void rational_test_already_reduced(struct kunit *test)
{
    unsigned long num, den;

    /* 3/7 is already in lowest terms */
    rational_best_approximation(3, 7, ULONG_MAX, ULONG_MAX, &num, &den);
    KUNIT_EXPECT_EQ(test, num, 3UL);
    KUNIT_EXPECT_EQ(test, den, 7UL);
}

static void rational_test_zero_numerator(struct kunit *test)
{
    unsigned long num, den;

    rational_best_approximation(0, 5, ULONG_MAX, ULONG_MAX, &num, &den);
    KUNIT_EXPECT_EQ(test, num, 0UL);
    /* den can be anything nonzero; just check num */
}

/* Declare the test suite */
static struct kunit_case rational_test_cases[] = {
    KUNIT_CASE(rational_test_basic),
    KUNIT_CASE(rational_test_already_reduced),
    KUNIT_CASE(rational_test_zero_numerator),
    {}
};

static struct kunit_suite rational_test_suite = {
    .name = "rational",
    .test_cases = rational_test_cases,
};

kunit_test_suite(rational_test_suite);

Understanding the difference between KUNIT_EXPECT_* and KUNIT_ASSERT_* is crucial for writing useful tests.

KUNIT_EXPECT_EQ(test, left, right) (and variants _NE, _LT, _LE, _GT, _GE, _NULL, _NOT_NULL, _TRUE, _FALSE, _STREQ, etc.) are non-fatal assertions. If the check fails, the test case is marked as failed and the failure is logged, but execution of the current test case continues. This is usually what you want: it allows a single test run to report multiple independent failures, giving you more diagnostic information.

KUNIT_ASSERT_EQ(test, left, right) is a fatal assertion. If it fails, the current test case immediately aborts via kunit_abort(). Use this when continuing after a failure makes no sense — for example, if a pointer returned by an allocator is NULL, there is no point dereferencing it in subsequent assertions.

Fixtures allow you to share setup/teardown logic across multiple tests in a suite. The struct kunit_suite has .init and .exit function pointers. .init is called before each test case; .exit is called after each test case (even if the test aborted). This guarantees cleanup.

For more complex shared state, KUnit supports struct kunit_suite with an .suite_init (called once before all cases) and .suite_exit (called once after all cases).

Resource-managed allocations are a killer feature. Any allocation made via kunit_kmalloc(test, size, GFP_KERNEL) is automatically freed when the test case ends. This prevents test-induced memory leaks and makes test teardown trivially correct.

/* Demonstrating fixtures and resource management */
#include <kunit/test.h>

struct my_test_priv {
    void *buffer;
    int   initial_count;
};

static int my_suite_init(struct kunit *test)
{
    struct my_test_priv *priv;

    /* kunit_kmalloc is automatically freed at test end */
    priv = kunit_kmalloc(test, sizeof(*priv), GFP_KERNEL);
    if (!priv)
        return -ENOMEM;  /* test is skipped if init fails */

    priv->buffer = kunit_kmalloc(test, 4096, GFP_KERNEL);
    if (!priv->buffer)
        return -ENOMEM;

    priv->initial_count = 42;
    test->priv = priv;  /* attach to test for retrieval in cases */
    return 0;
}

static void test_buffer_not_null(struct kunit *test)
{
    struct my_test_priv *priv = test->priv;

    /* Fatal: no point continuing if buffer is NULL */
    KUNIT_ASSERT_NOT_NULL(test, priv->buffer);

    /* Non-fatal: report all failures */
    KUNIT_EXPECT_EQ(test, priv->initial_count, 42);
    KUNIT_EXPECT_GE(test, priv->initial_count, 0);
}

static struct kunit_case my_cases[] = {
    KUNIT_CASE(test_buffer_not_null),
    {}
};

static struct kunit_suite my_suite = {
    .name  = "my_example",
    .init  = my_suite_init,
    /* no .exit needed — kunit_kmalloc handles cleanup */
    .test_cases = my_cases,
};
kunit_test_suite(my_suite);

TAP (Test Anything Protocol) is a simple, language-agnostic protocol for reporting test results as plain text. The kernel adopted TAP and extended it as KTAP to support nested test suites — something the original TAP13 spec does not handle well.

A KTAP result looks like this:

KTAP version 1
1..3
    KTAP version 1
    1..2
    ok 1 rational_test_basic
    not ok 2 rational_test_already_reduced
        # FAILED: expected den == 7 but got den == 1
        # rational_kunit_test.c:24
ok 1 rational
not ok 2 idr
not ok 3 bitmap

Parsing the output:

• 1..N is a plan line: "expect N test results."
• ok N name means test N passed.
• not ok N name means test N failed.
• Lines starting with # are diagnostic comments (failure details, skip reasons).
• # SKIP: reason inside a not ok block means the test was skipped, not failed.
• Nested KTAP blocks (indented) represent sub-suites.

Why does this matter? Because CI systems, the kernel robot (KernelCI), and tools like kunit.py parse KTAP to generate structured reports. When you run kunit.py run, it parses the KTAP stream from the UML kernel and shows you a colored summary. Understanding the raw format helps you debug CI failures and write custom reporting tools.

CONFIG_KUNIT_DEBUGFS exposes test results at /sys/kernel/debug/kunit/<suite>/results when running on real hardware, making it easy to retrieve results after boot.

The ksft_* macros in kselftest's kselftest.h similarly produce TAP output, keeping both kselftest and KUnit results parseable by the same tooling.

# Example: run KUnit tests and watch KTAP output live
# (from kernel source root, requires Python 3 and UML support)

# Run with verbose KTAP output:
./tools/testing/kunit/kunit.py run --raw_output 2>&1 | head -50

# On a booted kernel with CONFIG_KUNIT=y and CONFIG_KUNIT_DEBUGFS=y:
dmesg | grep -E '^(KTAP|ok |not ok|\s+(ok|not ok|#))'

# Or read from debugfs:
cat /sys/kernel/debug/kunit/rational/results

The best way to solidify KUnit knowledge is to walk through writing a real test from scratch. Let us test a simple kernel utility: roundup_pow_of_two() from include/linux/log2.h, which rounds an integer up to the next power of two.

Step 1: Find the function under test. roundup_pow_of_two(n) is defined as a macro/inline in include/linux/log2.h. It works for any unsigned long input.

Step 2: Identify the contract.

• For any n > 0, the result r satisfies r >= n and r is a power of two.
• For n already a power of two, r == n.
• For n == 1, result is 1.
• For n == 0, behavior is undefined (document this).

Step 3: Write the test file. By convention, place it adjacent to the code. For a header-only utility, a good home is lib/math/log2_kunit_test.c.

Step 4: Register with Kconfig. Add a config LOG2_KUNIT_TEST entry in lib/math/Kconfig and a line in lib/math/Makefile.

Step 5: Verify with kunit.py. Run ./tools/testing/kunit/kunit.py run 'log2*' to execute only the new suite.

This workflow — identify contract → write cases → register → run via UML — is the standard KUnit development loop. The UML kernel typically rebuilds and runs in under 30 seconds for small changes.

/* lib/math/log2_kunit_test.c */
#include <kunit/test.h>
#include <linux/log2.h>   /* roundup_pow_of_two */

static void log2_test_roundup_power_of_two(struct kunit *test)
{
    /* Already a power of two — must be unchanged */
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(1UL),   1UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(2UL),   2UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(4UL),   4UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(1024UL), 1024UL);

    /* Not a power of two — must round up */
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(3UL),   4UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(5UL),   8UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(1023UL), 1024UL);
    KUNIT_EXPECT_EQ(test, roundup_pow_of_two(1025UL), 2048UL);
}

static void log2_test_result_is_power_of_two(struct kunit *test)
{
    unsigned long inputs[] = { 1, 2, 3, 7, 100, 255, 256, 1000, 65535 };
    int i;

    for (i = 0; i < ARRAY_SIZE(inputs); i++) {
        unsigned long r = roundup_pow_of_two(inputs[i]);
        /* A power of two has exactly one bit set: r & (r-1) == 0 */
        KUNIT_EXPECT_EQ_MSG(test, r & (r - 1), 0UL,
            "input=%lu result=%lu is not a power of two",
            inputs[i], r);
    }
}

static struct kunit_case log2_test_cases[] = {
    KUNIT_CASE(log2_test_roundup_power_of_two),
    KUNIT_CASE(log2_test_result_is_power_of_two),
    {}
};

static struct kunit_suite log2_test_suite = {
    .name = "log2",
    .test_cases = log2_test_cases,
};
kunit_test_suite(log2_test_suite);

MODULE_LICENSE("GPL v2");

syzkaller is a coverage-guided kernel fuzzer developed by Google. It is, without exaggeration, the single most productive bug-finding tool in the history of the Linux kernel. As of 2024, syzkaller (via syzbot, Google's continuous fuzzing service) has found and tracked over 20,000 kernel bugs. Understanding how it works is essential for any serious kernel developer.

The core idea: coverage-guided fuzzing. A naive fuzzer generates random system call sequences and hopes to crash the kernel. This finds shallow bugs but quickly gets stuck. syzkaller is smarter: it uses KCOV (Kernel Coverage instrumentation) to measure which kernel code paths each test case exercises. When a new test case explores a previously unseen code path, it is kept in the corpus and mutated further. This drives the fuzzer deeper into the kernel, methodically exploring states that random inputs would never reach.

KCOV: CONFIG_KCOV=y instruments every basic block in the kernel with a PC-tracing hook. When a process makes system calls, KCOV records which kernel addresses were executed. This is exposed at /sys/kernel/debug/kcov — a special file that a process can mmap to get a per-syscall trace of kernel PCs. syzkaller reads this after each test case to measure coverage.

Syscall descriptions (sys/linux/*.txt): syzkaller does not generate truly random bytes. Instead, it uses a formal description language to describe the types, ranges, and relationships of syscall arguments. For example, the description for read(fd fd, buf buffer[out], count len[buf]) tells the fuzzer that the first argument should be an open file descriptor (from a pool of valid FDs it has created), the second should be a writable buffer, and the third should be the length of that buffer. This structured approach generates valid-enough inputs to reach deep kernel code, while still being adversarial.

The syzkaller architecture:

• syz-manager: runs on a host machine, manages a pool of VMs.
• syz-fuzzer: runs inside each VM, generates test programs, measures coverage, manages the corpus.
• syz-executor: a minimal, fast process inside each VM that actually executes the generated syscall sequences.
• syz-repro: takes a crash log and attempts to minimize the crashing program to a short, reproducible test case.

syzbot: Google's public continuous fuzzing service at syzkaller.appspot.com runs syzkaller 24/7 against linux-next and various stable branches. All crashes are published publicly. Each bug has a dashboard entry showing the crash log, the reproducer program, the git bisect result (which commit introduced the bug), and who is working on a fix.

# Minimal example: enable KCOV in a test kernel config
# Add to your .config or use 'make menuconfig':
# CONFIG_KCOV=y
# CONFIG_KCOV_ENABLE_COMPARISONS=y  (for comparison tracing)
# CONFIG_DEBUG_INFO=y               (for symbolization)
# CONFIG_KASAN=y                    (to catch memory errors)

# Read a basic KCOV trace from userspace:
cat << 'EOF' > /tmp/kcov_test.c
#include <stdio.h>
#include <stddef.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>

#define KCOV_INIT_TRACE     _IOR('c', 1, unsigned long)
#define KCOV_ENABLE         _IO('c', 100)
#define KCOV_DISABLE        _IO('c', 101)
#define COVER_SIZE          (64<<10)

int main(void) {
    int fd = open("/sys/kernel/debug/kcov", O_RDWR);
    ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE);
    uint64_t *cover = mmap(NULL, COVER_SIZE * sizeof(uint64_t),
                           PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    ioctl(fd, KCOV_ENABLE, 0);  /* start tracing */
    cover[0] = 0;               /* reset counter */
    read(0, NULL, 0);           /* one syscall to trace */
    ioctl(fd, KCOV_DISABLE, 0);
    uint64_t n = cover[0];
    printf("Covered %llu kernel PCs in one read() call\n", n);
    for (uint64_t i = 1; i <= n && i <= 10; i++)
        printf("  PC: 0x%llx\n", cover[i]);
    return 0;
}
EOF
gcc -o /tmp/kcov_test /tmp/kcov_test.c && /tmp/kcov_test

syzbot emails bug reports to the Linux kernel mailing list and posts them publicly at syzkaller.appspot.com. Knowing how to read one is a practical skill — you will encounter them in your inbox as a kernel developer.

Anatomy of a syzbot report:

1. Title and bug hash. e.g., KASAN: slab-use-after-free Read in fput. The title identifies the sanitizer that triggered (KASAN, UBSAN, KMSAN, lockdep, etc.) and the nature of the fault.

2. Bisect result. syzbot often identifies the exact commit that introduced the bug: Introduced by: commit abc123 ("mm: add new feature"). This is invaluable for the fix.

3. Crash log. The full dmesg output at the time of crash, including the KASAN report with the bad access address, the allocation/free stack traces, and the call stack at the point of the access.

4. Reproducer. The most valuable part. syz-repro has minimized the crashing syscall sequence to a short C program (syz-repro: ...) or a syzkaller program (prog:). A C reproducer can be compiled and run directly on a vulnerable kernel.

5. Config. The kernel .config used to reproduce the crash. This is critical because many bugs only manifest with specific sanitizers enabled.

Reproducing locally:

1. Check out the commit identified in the bisect (or linux-next).
2. Apply the config from the report (CONFIG_KASAN=y, etc.).
3. Build and boot in a VM (QEMU is standard).
4. Compile and run the C reproducer inside the VM.
5. Observe the crash in dmesg.

The fix lifecycle: Once you understand the bug, write a fix, add a regression test (usually a KUnit test or kselftest that would have caught the original bug), and submit both to the relevant mailing list. Reference the syzbot bug ID in your commit message.

Key sanitizers syzkaller relies on:

• KASAN (Kernel Address SANitizer): detects use-after-free, out-of-bounds accesses.
• KMSAN (Kernel Memory SANitizer): detects reads of uninitialized memory.
• UBSAN (Undefined Behavior SANitizer): detects signed overflow, invalid shifts, etc.
• lockdep: detects lock ordering violations and deadlocks statically at runtime.

syzkaller is most powerful when combined with runtime sanitizers that transform silent memory corruption into loud, informative crashes. Each sanitizer instruments the kernel at build time to detect a different class of bug.

KASAN (Kernel Address Sanitizer): Enabled with CONFIG_KASAN=y. Uses shadow memory to track the validity of each byte of kernel memory. On every load/store, a compile-time-inserted check reads the shadow byte. If the shadow indicates the memory is invalid (freed, out-of-bounds, stack-out-of-scope), KASAN prints a detailed report and optionally panics. Two modes:

• Generic KASAN: ~2x memory overhead, ~2x slowdown. Works everywhere.
• Software Tag-Based KASAN: uses pointer tagging (arm64 TBI). Lower overhead.
• Hardware Tag-Based KASAN: uses ARMv8.5-A MTE. Near-zero overhead in production.

KMSAN (Kernel Memory Sanitizer): Enabled with CONFIG_KMSAN=y (requires Clang). Tracks whether each bit of memory has been initialized. Catches bugs where uninitialized stack or heap data is leaked to userspace or used in security-sensitive comparisons. These bugs are often invisible without KMSAN.

UBSAN (Undefined Behavior Sanitizer): Enabled with CONFIG_UBSAN=y. Catches C undefined behavior: signed integer overflow, shift-out-of-bounds, misaligned accesses, null pointer dereferences (in some configs). Many "harmless" integer overflows in kernel code turn out to be security-relevant when UBSAN exposes them.

lockdep: The Lock Dependency Engine (CONFIG_PROVE_LOCKING=y) is unique — it is not a memory sanitizer but a dynamic lock-order analyzer. It builds a graph of all lock acquisition orderings at runtime. If it detects that the current acquisition would complete a cycle in this graph (a potential deadlock), it reports the problem immediately — without needing an actual deadlock to occur. lockdep has found hundreds of deadlock bugs in code paths that were extremely rarely exercised in practice.

The combination of KASAN + KMSAN + UBSAN + lockdep in a syzkaller VM means that bugs which would cause a silent hang or data corruption on production hardware manifest as loud, detailed, symbolized crash reports within milliseconds.

/* A bug that KASAN would catch: use-after-free */
#include <linux/slab.h>
#include <linux/module.h>

static int __init uaf_demo_init(void)
{
    char *buf = kmalloc(64, GFP_KERNEL);
    if (!buf)
        return -ENOMEM;

    strcpy(buf, "hello");
    kfree(buf);

    /* BUG: accessing freed memory */
    pr_info("%s\n", buf);  /* KASAN will fire here */

    return 0;
}

/* KASAN output would look like:
   BUG: KASAN: slab-use-after-free in uaf_demo_init+0x.../...
   Read of size 1 at addr ffff888... by task insmod/...
   Allocated by task insmod:
     kmalloc ...
     uaf_demo_init ...
   Freed by task insmod:
     kfree ...
     uaf_demo_init ...
*/
module_init(uaf_demo_init);
MODULE_LICENSE("GPL");

Understanding how testing fits into the patch contribution workflow is what separates a good kernel contributor from a great one.

Before submitting a patch:

1. Run make W=1 to check for compiler warnings. Many maintainers refuse patches that introduce new warnings.
2. Run scripts/checkpatch.pl --strict on your patch file. This checks coding style.
3. Run the relevant kselftests: make -C tools/testing/selftests/mm run_tests for an mm patch, etc.
4. If your change is in a subsystem with KUnit tests, run kunit.py run 'subsystem*' to ensure nothing is broken.
5. If you are adding a new feature, add tests. A patch adding a new API without tests will often get a comment: "can you add a KUnit test for this?"
6. Boot-test in a VM with KASAN and lockdep enabled. Many reviewers will ask "did you test with KASAN?" as a first question.

In the cover letter / commit message: Mention your test strategy. "I ran the mm kselftests, added a KUnit test in mm/foo_kunit_test.c, and boot-tested with KASAN on x86_64 and arm64" is a powerful statement of thoroughness.

The CI landscape:

• KernelCI (kernelci.org): builds and boots kernel trees on hundreds of real hardware boards. Automatically runs kselftests.
• 0day robot (Intel): automatically builds patches sent to LKML and reports build errors and some test failures.
• syzbot: continuously fuzzes linux-next. If your patch is in linux-next and introduces a bug, syzbot will likely find it within days.

Responding to syzbot reports: If syzbot files a bug against your recent patch, you should:

1. Reproduce it locally using the provided config and reproducer.
2. Understand the root cause.
3. Write a fix.
4. Add a regression test.
5. Send both fix and test to the mailing list with Reported-by: syzbot+<hash>@syzkaller.appspotmail.com in the commit message.
6. Send a #syz fix: commit subject reply to the syzbot email to link your fix to the report.

This workflow — test, submit, respond to CI, iterate — is the heartbeat of kernel development.

User Mode Linux is a port of the Linux kernel that runs as a regular userspace process on a host Linux system. When you run kunit.py run, it builds and boots a UML kernel — typically in under 30 seconds — runs all the KUnit tests, captures the output, and prints a summary. No VM required. No QEMU to configure. No physical hardware needed.

This matters enormously for development velocity. The standard kernel build-test-boot cycle on QEMU can take 5-10 minutes per iteration. With UML and KUnit, the cycle is typically 20-60 seconds. This makes test-driven development practical for kernel code in a way it has never been before.

Limitations of UML testing:

• UML cannot test architecture-specific code (x86 MSR access, ARM NEON intrinsics, etc.).
• UML cannot test device drivers that need real hardware or even full emulation.
• Some kernel subsystems have limited UML support.
• UML is single-core and has limited memory model fidelity.

For anything UML cannot cover, the next step is QEMU with virtme or virtme-ng, which boots a real kernel image in a VM in about 10-15 seconds and can run kselftests and KUnit tests on full x86_64 or arm64 kernels.

The development loop in practice:

# Edit code:
nano lib/math/some_function.c
nano lib/math/some_function_kunit_test.c

# Run tests (UML, fast):
./tools/testing/kunit/kunit.py run 'some_function*'

# Fix failures, repeat...

# When satisfied, run on real kernel with KASAN:
virtme-run --kdir . --mods=auto -- \
  python3 tools/testing/kunit/kunit.py run

This combination of UML for fast iteration and QEMU/virtme for sanitizer-enabled full testing covers the vast majority of kernel code correctness concerns before a patch even leaves your machine.