Undefined Behavior & the C Memory Model

The C standard defines three distinct categories of "things the standard does not fully determine," and confusing them is a common source of bugs.

Undefined Behavior (UB): The standard imposes no requirements whatsoever on what the implementation does. It is literally undefined — the compiler is allowed to assume it never happens, and if it does happen, anything can occur: the program can crash, produce wrong output, format your hard drive, or (most dangerously) appear to work correctly today and fail after the next compiler update. Classic examples: dereferencing a null pointer, signed integer overflow, accessing an array out of bounds, using an object after its lifetime ends (use-after-free).

Implementation-Defined Behavior (IB): The standard says the implementation must choose some behavior, document it, and then consistently do it. You can rely on it once you know what your specific compiler/platform does. Examples: the size of int (must be documented), whether char is signed or unsigned, the result of right-shifting a negative signed integer.

Unspecified Behavior: The standard allows a set of valid choices but does not require documentation or consistency. You cannot rely on it even within one program run. The classic example: the order in which function arguments are evaluated is unspecified — f(a++, a++) has unspecified behavior because the compiler can evaluate arguments in any order.

The key mental model: think of the C standard as a contract between you and the optimizer. UB is the clause that says "if you violate any of these rules, the contract is void and we can do anything." The optimizer exploits this: it proves you can't have UB (by the contract), therefore it can assume certain things never happen, therefore it can generate faster code. The optimization and the danger are the same thing.

// UB: signed overflow
int x = INT_MAX;
int y = x + 1;  // UB — optimizer assumes this never happens
                // GCC at -O2 may eliminate any check based on y>x

// IB: right-shift of negative value
int a = -8;
int b = a >> 1; // IB: could be -4 (arithmetic) or a large positive
                // Most x86/ARM compilers do arithmetic shift, but not guaranteed

// Unspecified: argument evaluation order
int i = 0;
printf("%d %d\n", i++, i++); // Unspecified — don't do this

Signed integer overflow is perhaps the most practically dangerous UB in systems code. The C standard says: if the mathematical result of a signed integer operation does not fit in the result type, the behavior is undefined. On virtually every real machine, the hardware wraps around using two's complement, and this is what most programmers expect to happen — but the compiler is not required to produce that result.

Why does the standard say UB instead of just mandating two's complement? Because in 1989, there were architectures (like ones' complement or sign-magnitude machines) where two's complement was not guaranteed. More importantly, declaring it UB gives the optimizer enormous freedom.

The canonical dangerous optimization: Loop trip-count analysis. If i is a signed int and you write for (int i = 0; i <= n; i++), the compiler knows (because overflow is UB) that i will never wrap around. It can therefore prove the loop iterates exactly n+1 times and hoist or vectorize accordingly. If you relied on i wrapping to terminate the loop, you have a bug.

The CVE pattern: A bounds check like if (a + b < a) { /* overflow detected */ } is dead code from the optimizer's perspective. Since a + b < a can only be true if overflow happened, and overflow is UB, and UB never happens (by assumption), the entire check is unreachable. The optimizer deletes it. The program then proceeds without the bounds check. Buffer overflow exploit follows.

C23 finally mandates two's complement representation for signed integers, but overflow remains UB — the representation is fixed but the behavior on overflow is still undefined to preserve optimization freedom. Use __builtin_add_overflow() (GCC/Clang) or <stdckdint.h> (C23) for safe arithmetic.

#include <limits.h>
#include <stdio.h>
#include <stdbool.h>

// WRONG: 'optimized away' at -O2
bool add_overflows_wrong(int a, int b) {
    return (a + b) < a;  // UB if a+b overflows; compiler deletes this check
}

// CORRECT: compare in unsigned domain (no UB)
bool add_overflows_ok(int a, int b) {
    return (unsigned)a + (unsigned)b > (unsigned)INT_MAX;
}

// CORRECT (GCC/Clang): use builtin
bool add_safe(int a, int b, int *result) {
    return __builtin_add_overflow(a, b, result); // returns true on overflow
}

int main(void) {
    int r;
    if (add_safe(INT_MAX, 1, &r))
        printf("overflow detected\n");
    else
        printf("result: %d\n", r);
    return 0;
}

Strict aliasing is the rule that says: a pointer of type T* may only be used to access an object of type T (with a small list of exceptions). Violating this rule is UB, and the consequence is that the compiler may assume two pointers of different types never alias (point to the same memory), and therefore it does not need to reload memory between accesses through them.

Why the rule exists: Without strict aliasing, the compiler must be extremely conservative. Every memory write through any pointer might have modified the object pointed to by any other pointer, so the compiler must reload from memory every time. This cripples optimization. With strict aliasing, the compiler can prove int* and float* don't overlap, so it can keep one in a register while writing through the other.

The classic violation (common in network code and type-punning):

float f = 3.14f;
int i = *(int*)&f;  // STRICT ALIASING VIOLATION — UB

The correct way to type-pun is with memcpy (the optimizer generates the same zero-cost code) or with a union (valid in C, not in C++).

Why the Linux kernel uses -fno-strict-aliasing: The kernel is full of legitimate type-punning — casting between struct page *, void *, and hardware register types. Rather than audit millions of lines for every aliasing assumption, Linus Torvalds decided it was safer to disable the optimization. If you're writing kernel code, you work under -fno-strict-aliasing, but you must understand why it exists when reading compiler output or porting code.

The exception list: char* and unsigned char* may alias anything. This is the blessed way to copy bytes. Also: a pointer to a struct may alias a pointer to its first member; a pointer to a signed T may alias unsigned T.

The restrict keyword (C99) goes in the opposite direction: it's a promise from the programmer that no other pointer aliases a given pointer for the duration of a scope. This lets the compiler optimize even more aggressively (think vectorized memcpy).

#include <string.h>
#include <stdio.h>
#include <stdint.h>

// WRONG: strict aliasing violation
float bad_bits_to_float(uint32_t u) {
    return *(float*)&u;  // UB: float* aliases uint32_t*
}

// CORRECT: memcpy — compiles to a single register move on x86/ARM
float good_bits_to_float(uint32_t u) {
    float f;
    memcpy(&f, &u, sizeof f);  // no UB; optimizer elides the copy
    return f;
}

// CORRECT in C: union (valid for type-punning in C, not C++)
float union_bits_to_float(uint32_t u) {
    union { uint32_t u; float f; } pun;
    pun.u = u;
    return pun.f;
}

// Demonstrating restrict: compiler can vectorize with this promise
void add_arrays(float * restrict dst,
                const float * restrict src,
                int n) {
    for (int i = 0; i < n; i++)
        dst[i] += src[i];  // SIMD-vectorized because no aliasing assumed
}

A sequence point is a point in the execution of a program at which all side effects of previous evaluations are guaranteed to be complete and no side effects of future evaluations have yet started. Between sequence points, the order of evaluation is unspecified (or in some cases, undefined).

C11 replaced the old "sequence point" terminology with the more precise "sequenced before" relation (a partial order on evaluations), but the practical meaning is the same for our purposes.

Why this matters: The expression i = i++ is UB in C (before C++17 and C11's specific rules), not just unspecified. The C standard says: if you modify an object and use it in the same expression without an intervening sequence point, it is UB. The classic trap is a[i] = i++: is i incremented before or after indexing?

Where sequence points occur (C11 "sequenced before" guarantees):

• After the left operand of &&, ||, , (the comma operator) is evaluated
• After the condition of ?: is evaluated
• At the end of a full expression (a statement)
• When calling a function: all argument evaluations are complete before the function body begins
• After a function returns: the return value is available before the calling expression continues

What is NOT sequenced: The order of evaluation of operands of most binary operators (+, -, *, =, [], etc.) is unspecified. The order of evaluation of function arguments is unspecified. This is why f(i++, i++) is undefined in C: both modifications happen in the same expression without a sequence point between them.

C11 improvement: C11 makes i = i + 1 well-defined (the right side is sequenced before the assignment) but i = i++ is still UB because the side effect of ++ is not sequenced with respect to the assignment to i.

The practical rule: never modify a variable more than once in a single expression, and never read a variable in the same expression where you write it, unless a sequencing operator (&&, ||, ,, ?:) separates them.

#include <stdio.h>

int main(void) {
    int i = 5;

    // UB: i modified twice with no sequence point between
    // i = i++;  // DO NOT DO THIS

    // Unspecified: which argument is evaluated first?
    // printf("%d %d\n", i++, i++);  // DO NOT DO THIS

    // WELL DEFINED: comma operator sequences left before right
    int a = (i++, i++);  // first i++ (i becomes 6), then i++ (i becomes 7)
                          // a = 6 (the value of the second i++)
    printf("a=%d i=%d\n", a, i);  // a=6, i=7

    // WELL DEFINED: short-circuit operators sequence
    int x = 0, y = 0;
    if (x++ > 0 && y++ > 0)  // x incremented; y NOT incremented (short-circuit)
        puts("both");
    printf("x=%d y=%d\n", x, y);  // x=1, y=0

    return 0;
}

Understanding that UB is a contract clause — not just "bad behavior" — requires seeing concretely how compilers exploit it. The optimizer does not just ignore UB; it actively uses the absence of UB as a proof to enable transformations.

The null-pointer deletion pattern: This is one of the most famous and dangerous:

void f(int *p) {
    *p = 42;        // (1)
    if (p == NULL)  // (2)
        abort();
}

The compiler sees: at point (1), p was dereferenced. Dereferencing a null pointer is UB. Since UB never happens (by assumption), p cannot be null at point (1). Therefore at point (2), p == NULL is always false. The compiler deletes the if branch. The null check that was supposed to protect you is gone.

The infinite loop pattern: Signed overflow is UB, so:

for (int i = 0; i >= 0; i++) { /* ... */ }

The compiler knows i starts non-negative and incrementing a non-negative int without overflow keeps it non-negative (since overflow is UB and thus impossible). Therefore i >= 0 is always true. This compiles to an infinite loop — not a loop that wraps to negative.

Tools to catch UB at runtime:

• clang -fsanitize=undefined (UBSan): inserts runtime checks for all UB the compiler knows about
• clang -fsanitize=address (ASan): catches buffer overflows, use-after-free
• clang -fsanitize=integer: catches signed overflow specifically

The kernel itself is compiled with UBSan in CI (CONFIG_UBSAN). If you are writing kernel patches, always run with sanitizers enabled.

Godbolt Compiler Explorer (godbolt.org) is an invaluable tool: paste suspect C code, compile with -O2, and watch for the branch that disappears. The absence of assembly is the tell.

// Compile with: gcc -O2 -o test test.c
// Then: clang -fsanitize=undefined -o test_san test.c

#include <stdio.h>
#include <stdlib.h>

// Demonstrates null-check elimination by optimizer
static void process(int *p) {
    *p = 42;           // dereference -> p cannot be null -> optimizer knows this
    if (p == NULL) {   // optimizer: always false, deletes this entire block
        fprintf(stderr, "null pointer!\n");
        exit(1);
    }
    printf("p=%p val=%d\n", (void*)p, *p);
}

int main(void) {
    int x;
    process(&x);   // fine
    // process(NULL); // would crash before the null check under -O2
    return 0;
}

// To see it: gcc -O2 -S -o test.s test.c
// Look for the null check in test.s — it may not be there.

The C standard defines objects: regions of storage. Every object has a lifetime (the interval during which storage is guaranteed to be reserved) and a type. Accessing memory outside an object's bounds, or after its lifetime ends, is UB.

Array out-of-bounds: int a[5]; a[5] = 0; is UB. The expression a[5] is defined as *(a + 5), and a + 5 is a valid pointer (one past the end is allowed for pointer arithmetic), but dereferencing it is UB. The compiler may generate code that appears to work but corrupts adjacent memory — including return addresses, function pointers, or security canaries.

Stack-frame lifetime: The classic dangerous pattern is returning a pointer to a local variable:

int *f(void) {
    int x = 42;
    return &x;  // x's lifetime ends when f returns
}              // caller gets a dangling pointer — UB to dereference

Modern compilers warn on this (-Wreturn-local-addr), but the kernel has many patterns involving stack-allocated buffers passed to callbacks; understanding lifetimes is essential.

Use-after-free: Accessing heap memory after calling free() is UB. The memory may be reused by the allocator immediately. This is one of the most common kernel security vulnerability classes (see use-after-free CVEs — there are hundreds).

Variable-Length Arrays (VLAs): C99 introduced VLAs (int a[n] where n is a runtime value). They are stack-allocated. They are banned in the Linux kernel (CONFIG_CC_NO_KERNEL_VLAS) because:

1. Stack overflow with no reliable detection
2. The size is attacker-controlled in many contexts
3. They complicate stack-frame analysis for security tools

Instead, use kmalloc/kzalloc for kernel allocations, or fixed-size arrays with explicit bounds checking.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// WRONG: dangling pointer to stack variable
int *bad_return(void) {
    int local = 99;
    return &local;  // WARNING: address of local variable returned
}                   // local is dead after return

// WRONG: use-after-free
void use_after_free_demo(void) {
    int *p = malloc(sizeof(int));
    *p = 42;
    free(p);
    // *p = 1;  // UB: accessing freed memory
    p = NULL;   // good practice: null out after free
}

// RIGHT: caller-owns allocation
int *good_alloc(void) {
    int *p = malloc(sizeof(int));
    if (!p) return NULL;
    *p = 99;
    return p;  // caller must free()
}

// RIGHT: in-bounds check before access
void safe_write(int *arr, size_t len, size_t idx, int val) {
    if (idx < len)      // unsigned comparison: idx >= 0 is always true
        arr[idx] = val;
}

int main(void) {
    int *p = good_alloc();
    if (p) {
        printf("%d\n", *p);
        free(p);
        p = NULL;
    }
    return 0;
}

Everything above assumed a single thread. The moment you have multiple threads, you face a second layer of non-determinism: memory ordering. The C11 standard introduced a formal memory model (borrowed heavily from the C++11 model) that defines precisely what values a thread is allowed to observe from memory.

Why the hardware doesn't just "do the obvious thing": Modern CPUs do not execute instructions in program order. They use:

• Store buffers: a write goes into a buffer before hitting cache; another thread may not see it yet
• Load reordering: a CPU may speculatively load from a later instruction before an earlier one completes
• Cache coherency delays: on ARM/POWER, a write that reaches L1 cache on one core may not be immediately visible to other cores

On x86, the memory model is relatively strong (Total Store Order, TSO). On ARM64 and RISC-V, it is much weaker — reorderings that are impossible on x86 are commonplace. Linux runs on all of these, so kernel code must be written to the weakest model that is correct everywhere.

The C11 model's key concept: happens-before: An operation A happens-before operation B if there is a causal chain ensuring B sees A's effects. You establish happens-before through:

• Program order within a single thread (always)
• atomic_store with memory_order_release paired with atomic_load with memory_order_acquire in another thread
• pthread_create/pthread_join
• mutex_lock/mutex_unlock

The memory orderings (<stdatomic.h>):

• memory_order_relaxed: no synchronization, just atomicity (use for counters)
• memory_order_acquire: this load "acquires" — all subsequent reads/writes in this thread see stores that were released before this point in the other thread
• memory_order_release: this store "releases" — all previous writes in this thread become visible to a thread that acquires
• memory_order_seq_cst (default): full sequential consistency — the strongest, most expensive, and what you should start with until you need performance

In the kernel: Linux does not use C11 atomics directly; it has its own atomic_t, READ_ONCE(), WRITE_ONCE(), smp_mb(), and smp_wmb() / smp_rmb() macros built on architecture-specific assembly. But they map directly to the concepts of release/acquire/full barriers. Understanding the C11 model is the mental foundation.

#include <stdatomic.h>
#include <threads.h>
#include <stdio.h>

// Producer/consumer with release-acquire semantics
static atomic_int data = 0;
static atomic_int ready = 0;

int producer(void *arg) {
    (void)arg;
    data = 42;  // (A) plain write — not atomic, but...
    // ...the release store below ensures (A) is visible before (B)
    atomic_store_explicit(&ready, 1, memory_order_release); // (B)
    return 0;
}

int consumer(void *arg) {
    (void)arg;
    int r;
    // Spin until ready: acquire load "synchronizes with" the release store
    while (!(r = atomic_load_explicit(&ready, memory_order_acquire))) // (C)
        ;  // busy-wait (simplified; real code uses futex/condvar)
    // The acquire at (C) synchronized with the release at (B),
    // so this thread is guaranteed to see the write to data at (A).
    printf("data = %d\n", data);  // guaranteed to print 42
    return 0;
}

int main(void) {
    thrd_t p, c;
    thrd_create(&p, producer, NULL);
    thrd_create(&c, consumer, NULL);
    thrd_join(p, NULL);
    thrd_join(c, NULL);
    return 0;
}
// compile: gcc -std=c11 -o mem mem.c -lpthread

A data race in C11 is formally defined as: two threads concurrently access the same memory location, at least one access is a write, and there is no synchronization between them that establishes a happens-before relationship. The result is UB — not just "wrong output," but the complete invalidation of the program's semantics.

This is a stronger statement than it sounds. A data race does not just mean "you might read a stale value." It means the compiler is allowed to assume data races don't exist and therefore optimize based on that assumption. A global variable read in a hot loop may be cached in a register — because the compiler sees no synchronization and therefore knows (by the no-data-race assumption) no other thread is writing it. If another thread does write it, the loop never sees the update.

volatile does NOT fix this: volatile prevents the compiler from caching a variable in a register — it forces every access to go through memory. This is useful for memory-mapped hardware registers, but it provides no inter-thread synchronization guarantees. Two threads using volatile int x still have a data race. volatile is a hardware-synchronization tool, not a thread-synchronization tool.

READ_ONCE and WRITE_ONCE in the kernel: Linux defines these macros (in include/linux/compiler.h) as wrappers around volatile casts. They prevent the compiler from tearing or coalescing accesses to shared variables that are protected by other synchronization (like spinlocks or RCU). They are not the same as atomics — they assume external synchronization provides the ordering guarantees. This is a subtle but crucial distinction.

Finding data races: ThreadSanitizer (clang -fsanitize=thread) is the tool of choice. The kernel has CONFIG_KCSAN (Kernel Concurrency Sanitizer), which instruments memory accesses to detect data races in the running kernel.

// DATA RACE: do not do this
#include <threads.h>
#include <stdio.h>

static int counter = 0; // shared, unprotected

int increment(void *arg) {
    for (int i = 0; i < 100000; i++)
        counter++;  // DATA RACE: read-modify-write, not atomic
    return 0;
}

// CORRECT: use C11 atomic
#include <stdatomic.h>
static atomic_int safe_counter = 0;

int safe_increment(void *arg) {
    for (int i = 0; i < 100000; i++)
        atomic_fetch_add_explicit(&safe_counter, 1, memory_order_relaxed);
        // relaxed is fine here: we only need the final value after join
    return 0;
}

// volatile does NOT protect against data races:
static volatile int vol_counter = 0;
// incrementing vol_counter from two threads is STILL UB
// (read-modify-write is not atomic even with volatile)

Knowing about UB is only useful if you have a systematic defense. Here is the practical toolkit.

Compile-time flags that reveal UB:

• -Wall -Wextra -Wpedantic: turn on all standard warnings; many UB situations produce warnings
• -Wundefined-behavior (Clang): warn specifically about statically detectable UB
• -ftrapv: make signed overflow trap at runtime (slow; use in testing)
• -fno-strict-aliasing: disable strict-aliasing optimizations (the Linux kernel's choice)
• -D_FORTIFY_SOURCE=2: adds buffer-overflow checks to libc functions

Runtime sanitizers (use during testing, not production — significant overhead):

• UBSan (-fsanitize=undefined): catches signed overflow, null deref, misaligned access, out-of-bounds (with bounds information), and more
• ASan (-fsanitize=address): catches heap/stack buffer overflows, use-after-free, use-after-return
• TSan (-fsanitize=thread): catches data races
• MSan (-fsanitize=memory): catches reads of uninitialized memory (Clang only)

Coding discipline checklist:

1. Never assume integers won't overflow — use __builtin_*_overflow or <stdckdint.h> for security-critical arithmetic
2. Use size_t for array indices and sizes — unsigned arithmetic wraps predictably
3. After free(), null the pointer immediately
4. Prefer memcpy for type-punning over pointer casts
5. Never return pointers to local variables
6. Use static_assert and assert liberally — they cost nothing in release builds
7. When in doubt, check Godbolt: compile with -O2 and see what the optimizer believes about your code

Kernel-specific: the kernel CI runs multiple sanitizer configurations (CONFIG_KASAN, CONFIG_UBSAN, CONFIG_KCSAN). Any patch submitted to Linus's tree that introduces sanitizer failures will be rejected. Run the kernel with make KCFLAGS="-fsanitize=undefined" in a QEMU VM before submitting patches.

#!/bin/bash
# Build a C file with all defensive flags for development
# Usage: ./build_safe.sh myfile.c

FILE="$1"
OUT="${FILE%.c}_safe"

clang \
  -std=c11 \
  -O1 \
  -Wall -Wextra -Wpedantic \
  -fsanitize=undefined,address \
  -fno-omit-frame-pointer \
  -g \
  -o "$OUT" "$FILE" \
  && echo "Built: $OUT" \
  || echo "Build failed"

# Run with full UBSan verbosity:
# UBSAN_OPTIONS=print_stacktrace=1 ./$OUT