Debugging C: GDB, Valgrind & Sanitizers

Before touching any tool, internalize the method. Debugging is applied science:

1. Observe: what does the program actually do?
2. Hypothesize: what single specific thing could explain it?
3. Predict: if your hypothesis is right, what else must be true?
4. Test: run an experiment that can falsify the hypothesis.
5. Repeat: your experiment either confirms, refutes, or refines the hypothesis.

The key word is falsify. A good experiment is one that, if your hypothesis is wrong, gives you a different result. This sounds obvious but most beginners skip it — they throw changes at a program hoping something sticks. That is not debugging; it is thrashing.

Printf debugging is the most primitive form of this loop: insert print statements, observe output, form hypothesis. It works, and experienced engineers use it constantly. Its limits are: you cannot pause execution and explore state you didn't think to print; you cannot replay execution; and you leave behind scaffolding that must be cleaned up.

Interactive debuggers (GDB) let you stop time, look at anything, and change state. Dynamic analysis tools (Valgrind, ASan) run your program in an instrumented environment that catches entire classes of bugs automatically. Neither replaces the scientific mindset — they just give you better instruments.

A real heuristic from kernel developers: when a bug is hard, the first thing you write is a minimal reproducible test case. Shrink the program until the bug still exists. Small programs are far easier to debug than large ones.

/* Minimal reproducible case discipline:
 * Before reaching for GDB, reduce the problem.
 * If a 5000-line program crashes, does this crash too?
 */
#include <stdio.h>
#include <string.h>

int main(void) {
    char buf[8];
    /* Deliberately minimal: does strcpy with len > 8 still segfault? */
    strcpy(buf, "Hello, World!");  /* overflow: yes it does */
    printf("%s\n", buf);
    return 0;
}

A compiled binary is a translation of your source code into machine instructions. By default, compilers optimize aggressively: they reorder instructions, eliminate variables, inline functions, and fold constants. All of that makes the binary faster but makes debugging nearly impossible — the debugger cannot map a machine instruction back to a source line because the mapping was destroyed.

The -g flag tells the compiler to emit debug information (usually in DWARF format on Linux) embedded in the binary. DWARF is a rich data format that records:

• Which machine instruction corresponds to which source line
• Where each local variable lives at each point (in a register? on the stack at offset -16?)
• The types of all variables (so GDB can display int *p properly)
• Call frame information (how to reconstruct the call stack)

-O0 disables all optimizations. This makes the compiled code almost a 1-to-1 translation of your source, so every source line does exactly one thing, variables are always in memory (not just registers), and stepping through code in GDB does what you expect.

-O2 -g is common in production: optimized code with debug info attached. Stepping through this in GDB is confusing (lines jump around, variables say <optimized out>) but it is still useful for post-mortem analysis.

Frame pointers: Modern compilers often omit the frame pointer register (rbp on x86-64) to free it for general use. This breaks stack unwinding in some tools. -fno-omit-frame-pointer restores it — Valgrind and some profilers benefit from it.

For development, always use: gcc -g -O0 -Wall -Wextra -fno-omit-frame-pointer your_file.c -o your_binary

# Compile with full debug info and no optimization
gcc -g -O0 -Wall -Wextra -fno-omit-frame-pointer buggy.c -o buggy

# Verify debug info is present
file buggy
# → buggy: ELF 64-bit LSB executable, ... with debug_info, not stripped

# See the DWARF debug sections
readelf -S buggy | grep debug
# → .debug_info   .debug_abbrev   .debug_line   .debug_str  ...

# Strip debug info for a release build (separate file for crash reporting)
objcopy --only-keep-debug buggy buggy.dbg
objcopy --strip-debug buggy
objcopy --add-gnu-debuglink=buggy.dbg buggy

GDB (GNU Debugger) is an interactive program that controls another program — the inferior — and lets you stop it at any point, inspect all of its state, and resume. Think of it as a time-travel microscope for running processes.

Starting GDB:

• gdb ./program — load the program, don't run it yet
• gdb ./program core — load a core dump for post-mortem
• gdb -p PID — attach to an already-running process
• Inside GDB: run arg1 arg2 or r starts the program

Breakpoints tell GDB to stop when execution reaches a specific location:

• break main — stop at the start of main
• break file.c:42 — stop at line 42 of file.c
• break function_name — stop at any call to that function
• break *0x401234 — stop at a specific address (useful when source isn't available)
• info breakpoints — list all breakpoints
• delete 2 — delete breakpoint number 2
• condition 3 x > 10 — only trigger breakpoint 3 when x > 10

Stepping through code:

• next (or n) — execute the next source line, skipping over function calls
• step (or s) — execute the next source line, stepping into function calls
• finish — run until the current function returns, then stop
• continue (or c) — resume execution until the next breakpoint
• stepi / nexti — step one machine instruction at a time

Examining state:

• print x (or p x) — print the value of variable x
• print *ptr — dereference and print
• print arr[0]@10 — print 10 elements of array arr starting at index 0
• display x — automatically print x every time execution stops
• x/10xw 0xaddr — examine memory: 10 items, hex format, word-sized
• info locals — all local variables in current frame
• info registers — all register values

The call stack:

• backtrace (or bt) — print the full call stack
• bt full — print stack with local variables at each frame
• frame 3 — switch to frame 3 (0 is innermost)
• up / down — move between frames

/* buggy.c — a simple use-after-free to debug */
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

void process(char *data) {
    printf("Processing: %s\n", data);
    free(data);          /* freed here */
}

int main(void) {
    char *buf = malloc(32);
    strcpy(buf, "hello");
    process(buf);
    printf("Length: %zu\n", strlen(buf)); /* use after free! */
    return 0;
}

/* GDB session:
$ gcc -g -O0 buggy.c -o buggy
$ gdb ./buggy
(gdb) break process
(gdb) run
Breakpoint 1, process (data=0x555555559260 "hello") at buggy.c:6
(gdb) print data
$1 = 0x555555559260 "hello"
(gdb) next          <- advances to free(data)
(gdb) next          <- free() executed, data pointer now dangling
(gdb) finish        <- return from process()
(gdb) print buf     <- buf still holds old (freed) address
$2 = 0x555555559260 ""   <- contents undefined after free
(gdb) backtrace
#0  main () at buggy.c:14
*/

Breakpoints stop at a location. But sometimes you know a variable is getting corrupted — its value changes to garbage — but you have no idea where in the code it happens. This is the classic memory corruption scenario, and breakpoints alone are useless.

Watchpoints solve this: they tell GDB to stop execution whenever a specific memory address is read or written. The hardware (x86 debug registers DR0–DR3, ARM watchpoint registers) monitors the address and generates a debug trap the instant it is accessed — with zero overhead until it triggers.

(gdb) watch global_var           # stop when global_var is written
(gdb) watch -l *ptr              # watch the memory ptr points to, not ptr itself
(gdb) rwatch arr[5]              # stop when arr[5] is READ
(gdb) awatch arr[5]              # stop on any access (read or write)

Hardware watchpoints are limited (x86-64 has 4 debug registers). When you have more than 4, GDB falls back to software watchpoints, which are extremely slow (it single-steps and checks after every instruction).

Conditional watchpoints combine the power of both:

(gdb) watch x
(gdb) condition 1 x < 0   # only trigger when x goes negative

Reverse debugging is a mind-bending capability: GDB can record every instruction your program executes and then play them backwards. You set a breakpoint on the symptom (the crash, the wrong value), then step backwards in time until you find the cause:

(gdb) record          # start recording execution
(gdb) continue        # run until crash/wrong value
(gdb) reverse-next    # go backwards one source line
(gdb) reverse-stepi   # go backwards one instruction

This is expensive in time and memory but invaluable for "how did this variable get this value?" questions. The kernel-mode equivalent is rr (Mozilla's record-and-replay tool), which is widely used by kernel developers.

/* corruption.c — a classic "who wrote this variable?" puzzle */
#include <stdio.h>

static int counter = 0;

void increment(void)  { counter++;         }
void reset_wrong(void){ counter = -999999; } /* the culprit */
void do_work(void)    { increment(); increment(); reset_wrong(); increment(); }

int main(void) {
    do_work();
    if (counter != 3)
        printf("BUG: counter=%d, expected 3\n", counter);
    return 0;
}

/* GDB watchpoint session:
$ gcc -g -O0 corruption.c -o corruption
$ gdb ./corruption
(gdb) start                    <- breaks at main
(gdb) watch counter            <- hardware watchpoint on counter
(gdb) continue
Hardware watchpoint 2: counter
Old value = 0
New value = 1
increment () at corruption.c:5
(gdb) continue
Old value = 1,  New value = 2  ...(at increment)
(gdb) continue
Old value = 2,  New value = -999999
reset_wrong () at corruption.c:6    <- caught!
*/

When a program crashes (SIGSEGV, SIGABRT, etc.), the OS can save its entire memory image to disk as a core dump — a binary file containing all memory, register values, and thread state at the moment of death. This is forensics: you examine the scene after the fact.

Enabling core dumps:

ulimit -c unlimited    # allow unlimited core file size (per-shell)
# Or permanently in /etc/security/limits.conf:
# *  soft  core  unlimited

On modern Linux with systemd, cores are handled by systemd-coredump and stored in the journal. Retrieve with coredumpctl list and coredumpctl gdb.

The core file path is controlled by /proc/sys/kernel/core_pattern. A typical setting:

echo '/tmp/cores/core.%e.%p' > /proc/sys/kernel/core_pattern
# %e = executable name, %p = PID

Loading a core in GDB:

gdb ./program core.12345
(gdb) bt              # see the backtrace at crash time
(gdb) frame 0         # examine the crashing frame
(gdb) info registers  # exact register state at crash
(gdb) x/20i $rip-20  # disassemble around the crash point

Reading a backtrace: This is a critical skill. A backtrace reads bottom-up (frame 0 is the innermost, deepest call). Example:

#0  0x00007f... in __memcpy_avx_unaligned ()
#1  0x000055... in process_packet (buf=0x0, len=1500) at net.c:88
#2  0x000055... in handle_input () at main.c:42
#3  0x000055... in main () at main.c:100

Frame 1 tells you: process_packet was called with buf=0x0 (a null pointer!), and it called memcpy on it. The bug is in frame 2 calling process_packet with a null argument.

Stripped binaries: Production binaries often have debug symbols stripped to reduce size. GDB will show only addresses. Solutions:

• Keep the .dbg file with symbols and load it: gdb -ex 'set debug-file-directory /path/to/dbg' ./stripped_bin core
• Use addr2line -e binary 0xdeadbeef to map addresses to source lines if you have the unstripped binary
• eu-addr2line from elfutils is more robust

/* crash.c — reliably produces a segfault for core dump practice */
#include <stdio.h>
#include <string.h>

typedef struct { int id; char name[16]; } Record;

Record *find_record(Record *table, int n, int id) {
    for (int i = 0; i < n; i++)
        if (table[i].id == id) return &table[i];
    return NULL;  /* not found */
}

int main(void) {
    Record table[] = {{1, "alice"}, {2, "bob"}};
    Record *r = find_record(table, 2, 99); /* id 99 not present */
    /* Forgot to check r != NULL: classic null-deref */
    printf("Found: %s\n", r->name);  /* SIGSEGV */
    return 0;
}

/* Practice:
$ gcc -g -O0 crash.c -o crash
$ ulimit -c unlimited
$ ./crash          <- crashes, produces core
$ gdb ./crash core
(gdb) bt
#0  0x... in main () at crash.c:17
(gdb) print r
$1 = (Record *) 0x0    <- null pointer, confirmed
(gdb) frame 0
(gdb) list             <- show source around crash line
*/

C gives you direct access to memory. It also gives you the full power to misuse it. Valgrind/Memcheck is a dynamic binary instrumentation tool: it translates every instruction in your program into instrumented code that tracks the validity and definedness of every byte in memory. It catches:

1. Invalid read/write — accessing memory you don't own (heap overflow, use-after-free, stack overflow past valid array bounds).
2. Use of uninitialised values — reading a variable before writing it. C never zeros memory by default; local variables contain stack garbage.
3. Heap memory leaks — allocating memory with malloc but never freeing it. Memcheck tracks every allocation and at program exit reports what wasn't freed.
4. Invalid free — calling free() on a pointer you didn't get from malloc(), or freeing twice.
5. Mismatched allocation/deallocation — using delete to free malloc-allocated memory (important in C++).

How it works (briefly): Valgrind runs your program in a synthetic CPU (JIT-compiled). It maintains two shadow memory bitmaps:

• Valid-address (A) bits: which addresses are currently mapped/accessible
• Valid-value (V) bits: which bytes contain defined (initialized) values

Every memory access checks the A bits. Every use of a value checks the V bits. This is why Valgrind slows programs by ~20x — it is running two programs simultaneously.

Reading Valgrind output:

==12345== Invalid read of size 4
==12345==    at 0x108748: main (buggy.c:15)
==12345==  Address 0x522d040 is 0 bytes after a block of size 40 alloc'd
==12345==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/...)
==12345==    by 0x108700: main (buggy.c:10)

The PID is 12345. You read 4 bytes at an address 0 bytes past the end of a 40-byte malloc. That is a one-past-the-end bug (reading arr[10] when arr has 10 elements).

Useful flags:

• --leak-check=full — show full details of every leak (not just summary)
• --show-leak-kinds=all — include reachable leaks (pointers still exist but memory not freed)
• --track-origins=yes — for uninitialized value errors, track where the uninitialized memory came from (slower but invaluable)
• --error-exitcode=1 — make Valgrind return non-zero on errors (useful in CI)
• --suppressions=file — ignore known false positives (system libraries often trigger these)

Critical limitation: Valgrind runs only on the program's heap. Stack overflows beyond the probed region and static buffer overflows (global arrays) are harder to catch. ASan (next concept) handles these better.

/* memleak.c — three common memory errors for Valgrind */
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

void uninit_read(void) {
    int x;             /* uninitialized */
    if (x > 0)         /* Valgrind: Conditional jump on uninitialised value */
        printf("positive\n");
}

void heap_overflow(void) {
    int *arr = malloc(10 * sizeof(int));
    arr[10] = 99;      /* one past the end: invalid write of size 4 */
    free(arr);
}

void memory_leak(void) {
    char *buf = malloc(64);
    strncpy(buf, "leaked!", 63);
    /* forgot free(buf) */
}

int main(void) {
    uninit_read();
    heap_overflow();
    memory_leak();
    return 0;
}

/* Run:
$ gcc -g -O0 memleak.c -o memleak
$ valgrind --leak-check=full --track-origins=yes ./memleak

Expect:
  - Conditional jump depends on uninitialised value(s)
  - Invalid write of size 4 (heap_overflow)
  - LEAK SUMMARY: 64 bytes in 1 block definitely lost
*/

Valgrind is powerful but slow (~20x). AddressSanitizer (ASan) is a compiler feature (supported by GCC and Clang) that instruments your code at compile time rather than interpreting it at runtime. The slowdown is typically only 2x, making it practical for large test suites.

Enable it: just add compiler flags:

gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer your.c -o your

(ASan works better with at least -O1 because that enables inlining needed to produce clean backtraces.)

How ASan works: It divides memory into application memory and shadow memory (1 byte of shadow for every 8 bytes of app memory, so 1/8 overhead). Every memory access is compiled into:

/* Original:  *ptr = value;  */
/* ASan-instrumented roughly: */
if (IsPoisoned(ptr))  ReportError(ptr);
*ptr = value;

Memory regions that are not valid to access are "poisoned" in the shadow: malloc unpoisons the allocated region; free re-poisons it. The stack frames, global arrays, and the space between heap allocations ("redzones") are always poisoned.

What ASan catches:

• Heap buffer overflow and underflow
• Stack buffer overflow
• Global buffer overflow
• Use-after-free
• Use-after-return (accessing a stack variable after the function returned — needs -fsanitize-address-use-after-return=always)
• Use-after-scope (accessing a variable outside its {...} block)
• Double-free and invalid-free
• Memory initialization order fiasco (C++)

ASan output is extremely clear:

==1234==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000028
WRITE of size 4 at 0x602000000028 thread T0
    #0 0x401234 in heap_overflow buggy.c:8
    #1 0x401300 in main buggy.c:15
...
Shadow bytes around the buggy address:
  0x602000000000: fa fa 00 00 00 00 00 00 00 00 fa fa ...
  →               fa = heap redzone, 00 = accessible, f8 = freed memory

The shadow byte legend at the bottom of ASan output tells you exactly what kind of memory the bad access landed in — a freed heap chunk (f8), a stack redzone (f1), a global redzone (f9), etc. Learn these codes.

ASan + Valgrind do not mix — never run both simultaneously. Use Valgrind for leak detection (ASan's leak detection is good but Valgrind's is more mature), use ASan for speed in large test suites.

/* asan_demo.c — stack overflow AND use-after-free, one file */
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

void stack_overflow(void) {
    char buf[8];
    /* Write 16 bytes into an 8-byte stack buffer */
    memset(buf, 'A', 16);   /* stack-buffer-overflow */
    printf("%c\n", buf[0]);
}

void use_after_free(void) {
    int *p = malloc(sizeof(int));
    *p = 42;
    free(p);
    *p = 99;   /* heap-use-after-free */
}

int main(void) {
    stack_overflow();
    use_after_free();
    return 0;
}

/* Build and run:
$ gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer asan_demo.c -o asan_demo
$ ./asan_demo

==1== ERROR: AddressSanitizer: stack-buffer-overflow on address ...
WRITE of size 16 at ...
    #0 in stack_overflow asan_demo.c:8
    #1 in main asan_demo.c:15

(after fixing stack_overflow, run again for use-after-free report)
*/

C's specification declares many operations as undefined behavior (UB): the standard says nothing about what happens, so compilers are free to assume they never occur and optimize accordingly. This creates a category of bugs that are silent and insidious — the program may appear to work on your machine but produce completely wrong results on another, or in a different optimization level, or after a seemingly unrelated code change.

Common UB examples:

• Signed integer overflow: INT_MAX + 1 — result is undefined (compilers assume it never happens and optimize loops on that basis)
• Null pointer dereference: technically UB before the access, allowing the compiler to remove null checks
• Shift overflow: 1 << 32 on a 32-bit int
• Out-of-bounds array access
• Misaligned pointer access: casting a char * to int * and dereferencing
• Floating point conversion overflow: converting a large double to int
• Using a NULL pointer (even before dereferencing, in some contexts)

UBSan is enabled with:

gcc -g -O1 -fsanitize=undefined your.c -o your
# Or combine with ASan (they are compatible):
gcc -g -O1 -fsanitize=address,undefined your.c -o your

UBSan output is a one-liner at the offending source location:

buggy.c:12:13: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
buggy.c:18:5: runtime error: null pointer passed as argument 1, which is declared to never be null
buggy.c:25:8: runtime error: shift exponent 32 is too large for 32-bit type 'int'

You can make UBSan abort immediately on the first error (useful in CI) with -fsanitize-undefined-trap-on-error or by setting the environment variable UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1.

The integer overflow UB gotcha in kernel code: The Linux kernel uses -fwrapv to make signed integer overflow wrap (like unsigned), opting out of the UB. Many kernel bugs historically came from UB optimizations removing bounds checks. UBSan is now mandatory in kernel CI builds.

/* ubsan_demo.c — a rogue's gallery of undefined behavior */
#include <stdio.h>
#include <limits.h>
#include <string.h>

void signed_overflow(void) {
    int x = INT_MAX;
    int y = x + 1;    /* UB: signed integer overflow */
    printf("%d\n", y);
}

void bad_shift(void) {
    unsigned int x = 1U << 31;  /* OK: unsigned */
    int y = 1 << 31;            /* UB on most platforms: shift into sign bit */
    (void)y;
    printf("%u\n", x);
}

void null_deref_ub(void) {
    int *p = NULL;
    /* UBSan catches this even before the dereference in some checks */
    int x = *p;   /* UB */
    printf("%d\n", x);
}

int main(void) {
    signed_overflow();
    bad_shift();
    /* null_deref_ub(); */   /* uncomment to test, will also crash */
    return 0;
}

/* Build:
$ gcc -g -O1 -fsanitize=undefined ubsan_demo.c -o ubsan_demo
$ UBSAN_OPTIONS=print_stacktrace=1 ./ubsan_demo

Output:
ubsan_demo.c:7:15: runtime error: signed integer overflow: 2147483647 + 1
ubsan_demo.c:12:16: runtime error: left shift of 1 by 31 places cannot be
  represented in type 'int'
*/

Raw GDB text output is functional but slow. Three features dramatically improve productivity:

1. TUI (Text User Interface): GDB has a split-screen terminal UI. Press Ctrl-X A inside GDB (or start with gdb -tui). You get a source window at the top showing the current line, and a command window at the bottom. Use layout regs to add a register panel, layout asm for disassembly. The arrow keys scroll the source view; GDB commands go in the lower pane.

2. ~/.gdbinit — your configuration file: GDB reads this file at startup. Essential settings:

set history save on
set history filename ~/.gdb_history
set print pretty on        # pretty-print structs
set print array on         # arrays on multiple lines
set disassembly-flavor intel   # Intel syntax (more readable than AT&T for x86)
set pagination off         # don't pause with "---Type <return>---"

3. Python pretty-printers: GDB embeds a Python interpreter. You can write Python scripts that teach GDB how to display complex data structures. The standard library ships pretty-printers for STL containers (C++) and there are community printers for Linux kernel data structures.

For kernel debugging, the scripts/gdb/ directory in the kernel source contains Python scripts that add GDB commands like lx-ps (list processes), lx-dmesg (print kernel log), and pretty-printers for struct list_head, struct task_struct, etc. You load them with:

(gdb) source /path/to/linux/scripts/gdb/vmlinux-gdb.py
(gdb) lx-ps

4. pwndbg and gef: Third-party GDB plugins popular in security research and low-level debugging. They add color, automatic context display (registers + disassembly + stack) on every stop, heap visualization, and many convenience commands. Install: pip install pwndbg or follow gef's one-liner installer.

# .gdbinit — a practical starter configuration
set history save on
set history filename ~/.gdb_history
set history size 10000
set print pretty on
set print array on
set print array-indexes on
set disassembly-flavor intel
set pagination off
set confirm off

# Useful aliases
define pf
  print *$arg0
end

define xxd
  set $i = 0
  while $i < $arg1
    printf "%02x ", *(unsigned char*)($arg0 + $i)
    set $i = $i + 1
  end
  printf "\n"
end
document xxd
  xxd addr len -- hex dump len bytes starting at addr
end

# Auto-load kernel scripts when debugging vmlinux
# add-auto-load-safe-path /path/to/linux/scripts/gdb

Printf debugging — inserting printf, fprintf(stderr, ...), or write() calls to trace execution — is not a sign of weakness. It is often the fastest tool for confirming a hypothesis quickly, and senior engineers use it constantly. The skill is knowing when to use it and how to do it cleanly.

When printf wins over GDB:

• The bug is timing-sensitive (a race condition): attaching GDB changes timing and the bug disappears (a Heisenbug)
• The program runs in an environment where GDB cannot attach (embedded target, kernel code running before scheduler init, inside an interrupt handler)
• You just need to confirm one specific value quickly and then move on
• The bug is in a tight loop that runs millions of times — interactive stepping is impractical

How to do it well:

1. Always print to stderr: fprintf(stderr, "[DEBUG] x=%d\n", x) — stdout may be buffered or redirected
2. Use __FILE__, __LINE__, __func__ for automatic location: fprintf(stderr, "%s:%d:%s: x=%d\n", __FILE__, __LINE__, __func__, x)
3. Build a debug macro that compiles to nothing in release builds:

#ifdef DEBUG
#define DBG(fmt, ...) fprintf(stderr, "[DBG %s:%d] " fmt "\n", \
                              __func__, __LINE__, ##__VA_ARGS__)
#else
#define DBG(fmt, ...) ((void)0)
#endif

4. Force flush after printing in crash scenarios: fflush(stderr) — if the program crashes immediately after printing, buffered output may be lost
5. In multi-threaded code, use dprintf(STDERR_FILENO, ...) — dprintf writes to a file descriptor directly and is async-signal-safe (safe inside signal handlers, write() is even safer)
6. In kernel code, use printk(KERN_DEBUG "...", ...) — this is the kernel's printf, always available

The trap to avoid: leaving debug printfs in production code. The macro approach makes it easy to strip them: compile with -DDEBUG for debug builds, without it for release. The Linux kernel's pr_debug() works exactly this way — it compiles to nothing unless DEBUG is defined or dynamic debug is enabled.

/* debug_macro.c — production-quality debug tracing */
#include <stdio.h>
#include <string.h>

/* This entire mechanism compiles to ZERO BYTES in release builds */
#ifdef DEBUG
  #define DBG(fmt, ...) \
    fprintf(stderr, "[DBG %s:%d] " fmt "\n", \
            __func__, __LINE__, ##__VA_ARGS__)
#else
  #define DBG(fmt, ...) ((void)0)
#endif

/* Also handy: assert-with-message (kept in all builds) */
#define ASSERT(cond, fmt, ...) do { \
    if (!(cond)) { \
        fprintf(stderr, "ASSERT FAILED %s:%d: " fmt "\n", \
                __FILE__, __LINE__, ##__VA_ARGS__); \
        __builtin_trap(); \
    } \
} while(0)

int divide(int a, int b) {
    DBG("divide called: a=%d b=%d", a, b);
    ASSERT(b != 0, "division by zero: a=%d", a);
    int result = a / b;
    DBG("result=%d", result);
    return result;
}

int main(void) {
    printf("%d\n", divide(10, 2));
    printf("%d\n", divide(7, 0));  /* triggers ASSERT */
    return 0;
}

/* Compile:
$ gcc -DDEBUG -g -O0 debug_macro.c -o debug_demo && ./debug_demo
[DBG divide:20] divide called: a=10 b=2
[DBG divide:23] result=5
5
ASSERT FAILED debug_macro.c:24: division by zero: a=7
*/

The sanitizers you learned — ASan, UBSan — work in userspace. The Linux kernel has its own parallel implementations that run inside the kernel:

KASAN (Kernel Address Sanitizer): Enabled with CONFIG_KASAN=y. Works identically to ASan but in kernel address space. It catches heap buffer overflows, use-after-free, and use-after-scope in kernel code. There are two modes:

• Generic KASAN: software shadow memory, ~1.5x memory overhead, ~2x slowdown. Runs on any architecture.
• Hardware tag-based KASAN: uses ARM MTE (Memory Tagging Extension) on ARMv8.5+. Zero shadow memory overhead; the hardware enforces memory tags. This is becoming the default for ARM64 kernel testing.

KUBSAN (Kernel UBSan): CONFIG_UBSAN=y. Catches the same UB in kernel C code as UBSan does in userspace. The kernel patches several known false positives.

syzkaller: An automated kernel fuzzer from Google that generates random syscall sequences and runs them on kernels built with KASAN+KUBSAN+KCOV (coverage instrumentation). It has found thousands of kernel bugs. The publicly reported bugs are at syzkaller.appspot.com.

Typical kernel developer workflow:

# Build kernel with all sanitizers
make defconfig
./scripts/config -e KASAN -e UBSAN -e KASAN_INLINE
make -j$(nproc)

# Boot in QEMU for testing
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
  -append "console=ttyS0 nokaslr" \
  -initrd /path/to/initramfs.cpio.gz \
  -nographic -m 2G

# KASAN report in dmesg:
# [ 123.456] BUG: KASAN: use-after-free in some_kernel_func+0x42/0x80
# [ 123.456] Read of size 8 at addr ffff888... by task kworker/...
# Call Trace:
#   dump_stack
#   print_address_description
#   kasan_report
#   some_kernel_func
#   ...

When you contribute to the kernel, your patch will be tested against this infrastructure automatically. A KASAN report in response to your patch means your patch introduced a memory bug — fix it before resubmitting.

/* Example KASAN report structure (annotated, not runnable):

[ 1234.567890] ==================================================================
[ 1234.567890] BUG: KASAN: slab-use-after-free in packet_rcv+0x1a4/0x5a0
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                              function + offset/total_size

[ 1234.567890] Read of size 8 at addr ffff88810a3b1e40 by task ksoftirqd/0/16
                          ^^^^^^                                ^^^^^^^^^^^^^^
                          access size                          task name/PID

[ 1234.567890] Allocated by task 1423:
[ 1234.567890]  __kmalloc+0x...
[ 1234.567890]  sk_prot_alloc+0x...
[ 1234.567890]  sk_alloc+0x...

[ 1234.567890] Freed by task 1423:
[ 1234.567890]  kfree+0x...
[ 1234.567890]  __sk_free+0x...
[ 1234.567890]  sk_free+0x...

/* The report tells you:
 * 1. WHERE the bad access happened (packet_rcv)
 * 2. WHAT was accessed (8 bytes at the address)
 * 3. WHERE the memory was allocated (sk_alloc)
 * 4. WHERE it was freed (__sk_free)
 * This is everything you need to find the lifetime bug.
 */