R0KernelPath
module 015 / 262

The Stack, Calling Conventions & Stack Frames

The call stack is the invisible scaffolding that makes functions work — understanding its exact layout, byte by byte, is the first step to seeing what compilers and CPUs really do.

~12h focused study12 concepts

Overview

Every time your C program calls a function, something remarkable happens: the CPU and compiler conspire to carve out a private workspace in memory, stash enough state to find the way home, execute the function, then tear it all down in microseconds — perfectly, millions of times per second. That workspace is the stack frame, and the rules that govern its creation and destruction are the calling convention.

This is not abstract theory. The stack is the reason recursion works, the reason local variables don't stomp on each other across nested calls, and the reason a buffer overflow can hijack a running program. Every performance optimization that involves "reducing function call overhead," every debugger backtrace you've ever read, every segfault you've ever hunted — all of it lives here.

In this module you will build a precise mental model of what happens during a function call at the level of machine instructions. You'll learn how the stack grows (it goes down, which surprises everyone), why arguments are passed the way they are, what the frame pointer is for, and how the compiler decides which registers to save. You'll read real x86-64 assembly produced by GCC, map it to C source line by line, and write your own programs that inspect and even deliberately overflow the stack. By the end, "stack frame" will not be a vague metaphor — it will be a concrete 64-byte (or 128-byte, depending) chunk of memory you can point to with a debugger.

Understanding this deeply is a prerequisite for reading kernel code fluently. The Linux kernel has its own calling conventions (asmlinkage, __fastcall, per-CPU stacks, interrupt stacks), and every context switch is literally a swap of stack pointers. The foundation you build here pays dividends all the way through to writing your first kernel patch.

why this matters for the kernel

Linux kernel code is littered with assembly trampolines, pt_regs structs (which are exactly a saved register file on the stack), alloca-style dynamic stack allocations, and ENTRY/RET macros that expand to calling-convention boilerplate. You cannot read arch/x86/kernel/entry_64.S or understand how copy_from_user protects the kernel stack without knowing, byte for byte, what a stack frame contains and where each field lives. The kernel also imposes a hard 8 KB stack limit per thread — understanding why that limit exists and what happens when it's exceeded (kernel oops, stack canary trips, CONFIG_VMAP_STACK guard pages) requires the exact mental model this module builds.

More practically, every debugging session you will ever have — in userspace or in the kernel — begins with a backtrace. A backtrace is nothing more than a walk of linked stack frames. Tools like gdb, perf, ftrace, bpftrace, and kgdb all depend on unwinding the stack correctly. When the unwind fails (because a frame pointer was omitted, or because an interrupt arrived mid-prologue), knowing the calling convention is the only way to recover. Investing the time to truly understand this material is one of the highest-leverage things a systems programmer can do.

interactive visualization
high address0x7fff…
main()ret0x... libcrbp—int n5SP·
low address ↓ grows0x7ffe…
return addresssaved base ptrparameterlocal var
▶ execute· step 1/7

Program starts. main()'s frame sits at the top of memory; SP points just below it.

main() → compute() → square(). Step through it: each call PUSHes a frame, each return POPs one — last in, first out. The stack pointer (SP) moves DOWNWARD as frames are added.

x86-64 Stack Frame Layout During a Function Call

rendering diagram…

Each box is a region within the current stack frame. Addresses decrease going downward. RBP points to the 'Saved RBP' slot; RSP points to the lowest allocated byte. The return address sits at [RBP+8], giving a debugger everything it needs to walk the call chain.

The concepts

01What Is the Stack and Why Does It Grow Downward?

The stack is a contiguous region of memory used for function call management. Unlike the heap (which grows upward from low addresses) or the text/data segments, the stack typically starts at a high virtual address and grows toward lower addresses with each push.

Why downward? Historical accident, but now a deeply entrenched convention on x86, x86-64, ARM, RISC-V, and nearly every modern architecture. The CPU's push instruction on x86 decrements the stack pointer (RSP) first, then stores the value — so 'pushing' means moving toward address zero.

Think of it like a stack of plates in a cafeteria, but the plates are labeled with addresses. The first plate placed has the highest address (say 0x7FFF_0000). Each new plate goes on top with a lower address (0x7FFE_FFF8, 0x7FFE_FFF0, ...). The top of the stack is always the lowest current address.

On Linux, each thread gets its own stack. For user-space threads, the default is 8 MB (soft limit; see ulimit -s). For kernel threads, it is a tiny 8 KB or 16 KB (architecture-dependent) — which is why the kernel has strict rules about not putting large arrays on the stack.

The stack pointer register (RSP on x86-64, SP on ARM/AArch64, SP on RISC-V) always points to the top of the stack — the last thing pushed, i.e., the item at the lowest current address.

// Visualise stack direction in C
#include <stdio.h>
void inner(void) {
    int x = 42;
    printf("inner &x = %p\n", (void*)&x);
}
void outer(void) {
    int y = 99;
    printf("outer &y = %p\n", (void*)&y);
    inner();
}
int main(void) {
    int z = 7;
    printf("main  &z = %p\n", (void*)&z);
    outer();
    return 0;
}
// Typical output (addresses decrease as we go deeper):
// main  &z = 0x7ffc5a3b1abc
// outer &y = 0x7ffc5a3b1a7c   <- lower than main's local
// inner &x = 0x7ffc5a3b1a3c   <- lower still

02Stack Frame Layout: The Exact Anatomy

A stack frame (also called an activation record) is the region of stack memory dedicated to one invocation of a function. It contains everything that function needs to run and return cleanly:

  1. Return address — where execution must resume after this function returns. Pushed automatically by the call instruction on x86-64.
  2. Saved frame pointer (RBP) — the previous frame's base, saved so the callee can restore it. (Omitted with -fomit-frame-pointer, discussed later.)
  3. Saved callee-saved registers — registers the callee promises to restore (RBX, R12-R15 on x86-64).
  4. Local variables — the function's own int, char[], structs, etc. The compiler assigns each a fixed offset from RBP (or RSP).
  5. Padding / alignment — the ABI requires RSP to be 16-byte aligned before a call. The compiler inserts padding to maintain this.
  6. Space for outgoing arguments — if the function calls others and those calls need stack-passed arguments (beyond the 6 register slots on x86-64), the compiler reserves space here in the caller's frame.

Here is the typical x86-64 frame layout (addresses decrease downward):

High addresses (caller's territory)
+-------------------------+
| ...caller's locals...   |
+-------------------------+  <-- caller's RBP (old frame base)
| return address          |  pushed by CALL instruction
+-------------------------+  <-- RSP after CALL, RBP after PUSH RBP
| saved RBP               |  callee stores old RBP here
+-------------------------+  <-- new RBP = RSP after MOV RBP,RSP
| saved callee-saves      |  RBX, R12, R13, R14, R15 if used
+-------------------------+
| local variables         |  [RBP - 8], [RBP - 16], ...
+-------------------------+
| alignment padding       |
+-------------------------+  <-- RSP (stack pointer)
Low addresses

The frame pointer (RBP) provides a stable reference: regardless of how much RSP moves during the function, [RBP - N] always refers to the same local variable.

// C source
int add(int a, int b) {
    int result = a + b;
    return result;
}

// Typical x86-64 GCC output (gcc -O0 -fno-omit-frame-pointer)
// add:
//   push   rbp              ; save caller's frame pointer
//   mov    rbp, rsp         ; establish new frame pointer
//   mov    DWORD PTR [rbp-20], edi  ; spill arg a (was in EDI)
//   mov    DWORD PTR [rbp-24], esi  ; spill arg b (was in ESI)
//   mov    edx, DWORD PTR [rbp-20]
//   mov    eax, DWORD PTR [rbp-24]
//   add    eax, edx
//   mov    DWORD PTR [rbp-4], eax   ; store into 'result'
//   mov    eax, DWORD PTR [rbp-4]   ; load return value into EAX
//   pop    rbp              ; restore caller's frame pointer
//   ret                     ; pop return address -> jump

03The Call and Ret Instructions: The Mechanism Behind Function Calls

On x86-64, the entire machinery of function calls reduces to two instructions:

CALL target does two things atomically:

  1. Pushes the address of the next instruction (the return address) onto the stack (RSP -= 8; [RSP] = next_ip).
  2. Jumps to target.

RET does the reverse:

  1. Pops the return address from the stack (next_ip = [RSP]; RSP += 8).
  2. Jumps to that address.

That's it. The whole elaborate frame construction (pushing RBP, subtracting from RSP for locals) is done by the compiler-generated function prologue, not by the CPU's call instruction. The CPU only deals in the return address.

This has a profound implication: if an attacker can overwrite the return address stored on the stack (the classic stack-smashing attack), they can redirect execution anywhere. This is the root cause of the most famous class of security vulnerabilities in history — stack buffer overflows.

On RISC-V, the equivalent is JAL ra, target (Jump And Link — saves return address in register ra) and JALR zero, ra, 0 (jump to ra). On AArch64, it's BL target (Branch with Link, saves to x30/lr) and RET (branches to lr).

The key difference: RISC architectures pass the return address in a register rather than pushing it to the stack, giving them one less memory operation on the call path.

// Demonstrate return address on stack
// WARNING: educational only, not portable across compilers/ABIs
#include <stdio.h>
#include <stdint.h>

void show_return_addr(void) {
    // On x86-64 with frame pointer:
    // [rbp+0] = saved rbp
    // [rbp+8] = return address
    uintptr_t *fp = (uintptr_t*)__builtin_frame_address(0);
    printf("saved RBP     = %p\n", (void*)fp[0]);
    printf("return address = %p\n", (void*)fp[1]);
}

int main(void) {
    show_return_addr();
    return 0;
}
// Compile: gcc -O0 -fno-omit-frame-pointer ret_addr.c -o ret_addr
// The return address printed will be an address inside main(),
// right after the call to show_return_addr().

04Calling Conventions: The ABI Contract

A calling convention is an agreement between caller and callee about how to pass arguments, where to put the return value, and which registers each side is responsible for preserving. Without this shared contract, separately-compiled translation units could not interoperate.

On Linux x86-64, the dominant convention is the System V AMD64 ABI:

Argument passing:

  • Integer/pointer args 1-6 go in registers: RDI, RSI, RDX, RCX, R8, R9 (in that order).
  • Floating-point args 1-8 go in XMM0-XMM7.
  • Additional args (7th integer and beyond) are pushed onto the stack, right-to-left order.
  • Structs larger than 16 bytes are passed by reference (caller allocates space, passes a pointer).

Return values:

  • Integer/pointer return value: RAX (and RDX for 128-bit values).
  • Floating-point: XMM0.

Register ownership (the most important thing to memorize):

  • Caller-saved (volatile): RAX, RCX, RDX, RSI, RDI, R8, R9, R10, R11. The callee may trash these freely. If the caller needs them after the call, the caller must save them first.
  • Callee-saved (non-volatile): RBX, RBP, R12, R13, R14, R15. The callee must save and restore these if it uses them.

Why this split? Caller-saved registers are naturally the argument and return-value registers — they get clobbered on every call by design. Callee-saved registers are typically loop counters, pointers to long-lived data structures, things the caller really needs to persist across a deep call tree.

On AArch64 (ARM 64-bit, the AAPCS64 ABI):

  • Arguments: x0-x7 (integers), v0-v7 (floats).
  • Return value: x0 (and x1 for 128-bit).
  • Callee-saved: x19-x28, x29 (frame pointer), x30 (link register — must save if you make further calls).
  • On RISC-V (RISC-V calling convention): a0-a7 for args/return, s0-s11 callee-saved, ra (x1) is the return address register.
// Illustrate register passing — watch the assembly
#include <stdio.h>

long sum6(long a, long b, long c, long d, long e, long f) {
    return a + b + c + d + e + f;
}

long sum7(long a, long b, long c, long d, long e, long f, long g) {
    return a + b + c + d + e + f + g; // 'g' goes on the stack!
}

// Compile and inspect:
// gcc -O0 -S calling_conv.c -o calling_conv.s
//
// In sum6: args arrive in rdi,rsi,rdx,rcx,r8,r9 — no stack needed
// In sum7: the call site does 'push <g_value>' before CALL,
//          and the callee reads it at [rbp+16]

05The Frame Pointer and Stackless Frames: -fomit-frame-pointer

The frame pointer (RBP on x86-64) is a convenience register that points to the base of the current frame. It provides a stable anchor: local variable result is always at [rbp-4] even if the function calls alloca() and RSP moves around dynamically.

But RBP is also a callee-saved register that the compiler has to push and pop — that's 2 extra instructions and 8 extra bytes per frame. On a tight hot loop with thousands of tiny function calls, this adds up. For this reason, GCC's -O2 and above enable -fomit-frame-pointer by default on x86-64: the compiler uses RSP-relative addressing ([rsp+12]) for all locals, eliminating RBP.

The trade-off: without a chain of saved RBPs, the debugger/profiler cannot easily walk the call stack. Tools like gdb, perf record, and ftrace need to unwind frames to generate backtraces. The solution is DWARF CFI (Call Frame Information) — compiler-generated metadata embedded in the .eh_frame ELF section that describes how to unwind each instruction. DWARF is powerful but complex; frame-pointer unwinding is simpler and more reliable at runtime (used heavily in eBPF, perf --call-graph fp).

The Linux kernel itself is built with -fno-omit-frame-pointer in several configurations precisely so that in-kernel profilers (perf, ftrace) can walk the kernel stack cheaply without DWARF.

Practical advice: for day-to-day development, add -fno-omit-frame-pointer to your CFLAGS. For production performance-critical builds, profile first — the overhead is often under 1%.

# See the difference yourself:
gcc -O2 -S example.c -o with_omit.s      # frame pointer omitted
gcc -O2 -fno-omit-frame-pointer -S example.c -o no_omit.s
diff with_omit.s no_omit.s
# You'll see 'push rbp' / 'mov rbp,rsp' / 'pop rbp' appear
# in no_omit.s but not in with_omit.s

# Or use Compiler Explorer (godbolt.org) — paste your C,
# toggle -fno-omit-frame-pointer, watch the prologue change live.

06Local Variables, Alignment, and the Red Zone

When a function declares local variables, the compiler calculates their total size at compile time and subtracts that amount from RSP in the function prologue (e.g., sub rsp, 48). Each variable gets a fixed offset from RBP (or RSP).

Alignment rules: The System V AMD64 ABI requires RSP to be 16-byte aligned at the point of every CALL instruction. Since CALL pushes 8 bytes (the return address), RSP is 8-byte aligned at the start of the callee's prologue. After push rbp (another 8 bytes), RSP is 16-byte aligned again — then the compiler subtracts a multiple of 16 for locals. If the total is not a multiple of 16, the compiler pads. This is why you often see sub rsp, 32 for a function with only a single int — 4 bytes of variable plus 12 bytes of padding.

Alignment matters because SIMD instructions (SSE, AVX) require 16- or 32-byte aligned memory access. Misaligned loads on modern CPUs are either slow (cross cache line) or illegal (some SSE instructions fault).

The red zone: On x86-64 Linux (System V ABI), leaf functions (functions that do not call other functions) can use the 128 bytes below RSP without adjusting RSP at all. This region is called the red zone. Signal handlers and interrupts are guaranteed not to clobber it. This optimization eliminates the sub rsp, N / add rsp, N sequence for small leaf functions.

Important: the kernel does not use the red zone (it uses -mno-red-zone) because hardware interrupts can arrive at any moment and would overwrite that memory.

// Leaf function that exploits the red zone
// gcc -O0 produces:
//   push rbp
//   mov  rbp, rsp
//   ; locals at [rbp-4], [rbp-8] -- within red zone area
//   ; NO sub rsp, N needed for a simple leaf!
int leaf(int x) {
    int doubled = x * 2;   // at [rbp-4], within 128-byte red zone
    return doubled;
}

// Check alignment:
void alignment_demo(void) {
    // After CALL: rsp % 16 == 8  (return addr pushed)
    // After PUSH RBP: rsp % 16 == 0
    // After SUB RSP, 16: rsp % 16 == 0  (still aligned)
    char buf[10];  // compiler rounds up to 16 for alignment
    // actual allocation: sub rsp, 16  (not sub rsp, 10)
    (void)buf;
}

07Recursion and the Stack: Cost, Depth, and Tail Calls

Every recursive call creates a new stack frame. For a recursion of depth N, you have N frames alive simultaneously, each consuming memory. This is the direct cost of recursion that iteration avoids.

Stack depth and overflow: On Linux, the default user stack is 8 MB. A typical frame might be 32-128 bytes. That gives you roughly 65,000-250,000 recursive calls before overflow. For most practical recursion (tree depth <= 50, quicksort with log N depth) this is fine. But naive recursive Fibonacci on large inputs, or parsing deeply nested data, can blow the stack.

Tail call optimization (TCO): A tail call is a function call that is the very last thing a function does — its result is returned directly without further computation. The key insight: the current frame is dead at the point of the tail call. The compiler can reuse the current frame instead of allocating a new one, turning recursion into a loop. GCC/Clang perform TCO at -O2 and above. You can verify with -S — a tail-recursive function compiled with TCO becomes a jmp (jump) instead of a call.

Mutual recursion and the kernel: The Linux kernel forbids deep recursion in interrupt handlers and soft-IRQ contexts because the interrupt stack is tiny (as small as 4 KB on some configurations). Deep filesystem recursion (path_openat → do_filp_open → ...) is limited by the VFS design to avoid blowing the kernel stack. This is a real constraint that influences kernel API design.

Fibonacci: the canonical example. Recursive Fibonacci is O(2^N) in time AND it allocates O(N) stack frames simultaneously (depth of the call tree). A loop uses O(1) stack.

#include <stdio.h>

// Naive recursive — O(2^n) time, O(n) stack frames
long fib_recursive(int n) {
    if (n <= 1) return n;
    return fib_recursive(n-1) + fib_recursive(n-2); // NOT a tail call
}

// Tail-recursive version — compiler can optimise away the stack growth
long fib_tail(int n, long a, long b) {
    if (n == 0) return a;
    return fib_tail(n-1, b, a+b); // tail call: last operation is the call
}

// To confirm TCO, compile with:
// gcc -O2 -S fib.c -o fib.s
// In fib.s, fib_tail's recursive call becomes a 'jmp fib_tail'
// (or the whole function becomes a loop with no call at all)

int main(void) {
    printf("fib(10) = %ld\n", fib_recursive(10));
    printf("fib(10) = %ld\n", fib_tail(10, 0, 1));
    return 0;
}

08Stack Overflow: What Actually Happens

A stack overflow is not a crash caused by an overflow of data into a neighbor variable (that's a buffer overflow). A stack overflow is when the stack pointer moves below the bottom of the stack's allocated virtual memory region.

On Linux, the stack region is backed by a guard page — a page of virtual memory marked with no read/write/execute permissions. When the stack pointer crosses into this page (either by a large function frame or by unbounded recursion), the next access triggers a segmentation fault (SIGSEGV). The kernel delivers SIGSEGV to the process, which terminates.

You can observe this: ulimit -s unlimited removes the soft stack limit, but the hard limit remains. Try void overflow(void) { char buf[100000]; overflow(); } and watch it segfault.

For the kernel: The kernel thread stack uses VMAP_STACK (enabled by CONFIG_VMAP_STACK) — each stack is virtually mapped with guard pages above and below. When the kernel stack overflows (common cause: deeply recursive VFS paths, or bug in a driver), the guard page triggers an NMI, and the kernel oops with STACK GUARD PAGE FAULT. Without VMAP_STACK, stack overflow was a silent corruption bug — the stack would overwrite adjacent kernel data structures. VMAP_STACK made this reliably detectable.

Stack canaries (GCC's -fstack-protector) are a different mechanism: a secret random value placed between local variables and the return address. Before ret, the compiler checks that the canary is unchanged. If a buffer overflow overwrote the return address, it also overwrote the canary, and the function aborts instead of jumping to attacker-controlled code.

#include <stdio.h>
#include <stdlib.h>

// WARNING: this program will crash with SIGSEGV
// Run it to observe a controlled stack overflow
static int depth = 0;
void recurse(void) {
    char waste[4096]; // each frame consumes 4 KB
    waste[0] = depth; // prevent optimisation away
    depth++;
    if (depth % 100 == 0)
        printf("depth=%d\n", depth); // watch it count up then die
    recurse();
}

int main(void) {
    recurse(); // will segfault, typically around depth 1000-2000
    return 0;
}
// Compile and run:
// gcc -O0 stack_overflow.c -o stack_overflow && ./stack_overflow
// Output: depth=100 ... depth=1700 ... Segmentation fault (core dumped)
//
// Check with stack protector:
// gcc -O0 -fstack-protector-all stack_overflow.c -o sp && ./sp

09Reading Real Assembly: Function Prologue and Epilogue

The prologue is the sequence of instructions at the start of a function that sets up the stack frame. The epilogue is the sequence at the end that tears it down. Recognizing these patterns is a core skill for reading disassembly, debugging, and understanding optimization.

Standard x86-64 prologue (with frame pointer, -O0):

push  rbp          ; save caller's frame pointer
mov   rbp, rsp     ; new frame base = current top of stack
sub   rsp, N       ; allocate N bytes for locals

Standard epilogue:

leave              ; equivalent to: mov rsp,rbp; pop rbp
ret                ; pop return address and jump

(leave is a single instruction that restores RSP from RBP and pops the saved RBP.)

Optimized prologue (with -O2, -fomit-frame-pointer):

sub   rsp, N       ; just allocate locals, no RBP bookkeeping

Epilogue:

add   rsp, N       ; deallocate locals
ret

AArch64 prologue (AAPCS64, GCC -O0):

stp   x29, x30, [sp, #-16]!  ; push frame pointer + link register
mov   x29, sp                ; set frame pointer
sub   sp, sp, #32            ; allocate locals

(stp = store pair; the ! means pre-decrement SP. Note: link register x30 is saved here because this is a non-leaf function.)

Tools to see this live: gcc -S -O0 file.c, objdump -d ./binary, or paste C into godbolt.org and select x86-64 gcc or ARM64 gcc.

# Quick workflow to read your own function's assembly
# 1. Write a small C file
cat > /tmp/demo.c << 'EOF'
int square(int x) { return x * x; }
int cube(int x)   { return x * square(x); }
EOF

# 2. Compile to assembly (AT&T syntax, Intel syntax with -masm=intel)
gcc -O0 -fno-omit-frame-pointer -S /tmp/demo.c -o /tmp/demo.s
cat /tmp/demo.s

# 3. Compile and disassemble the binary
gcc -O0 -fno-omit-frame-pointer /tmp/demo.c -o /tmp/demo
objdump -d -M intel /tmp/demo | grep -A 20 '<square>'

# 4. Try with optimisation on, see the difference
gcc -O2 -S /tmp/demo.c -o /tmp/demo_opt.s
# Notice: cube() may inline square() entirely, no call instruction

10Inspecting the Stack with GDB

The GNU Debugger is the most direct way to observe stack frames live. A few essential commands:

  • backtrace (or bt) — print all active frames, from innermost to outermost.
  • frame N — select frame N.
  • info frame — show the selected frame's registers: saved RBP, return address, local variables' addresses.
  • info locals — print all local variables in the selected frame.
  • x/16gx $rsp — dump 16 8-byte words at RSP (examine the raw stack).
  • p/x $rbp — print current frame pointer.
  • p/x *(void**)($rbp+8) — manually read the return address.

The workflow: set a breakpoint inside a deeply nested function, run to it, and then use bt to see the entire chain. Then use frame to jump between frames, and info locals to see each function's private variables.

This is exactly what the kernel's kgdb does — same commands, same frame walking — except now applied to kernel code running on a second machine or in QEMU.

For automated stack walking (e.g., in perf, bpftrace, eBPF programs), the tool reads RBP, follows the chain of saved RBPs (each saved RBP points to the previous frame's RBP), and reads the return address at [saved_rbp + 8]. This is the frame-pointer chain walk, and it's why -fno-omit-frame-pointer is so valuable for profiling.

# GDB session example (run this yourself)
gcc -O0 -g -fno-omit-frame-pointer /tmp/demo.c -o /tmp/demo_dbg

# Then in gdb:
# $ gdb /tmp/demo_dbg
# (gdb) break cube
# (gdb) run
# Breakpoint 1, cube (x=5) at demo.c:2
# (gdb) backtrace
# #0  cube (x=5) at demo.c:2
# #1  0x00005555555551a2 in main () at demo.c:8
# (gdb) info frame
# Stack level 0, frame at 0x7fffffffe370:
#  rip = 0x555555555149 in cube; saved rip = 0x5555555551a2
#  called by frame at 0x7fffffffe390
#  Arglist at 0x7fffffffe360, args: x=5
# (gdb) x/4gx $rsp
# 0x7fffffffe350: 0x00007fffffffe370  0x00005555555551a2
#                 ^saved rbp           ^return address

11Variadic Functions and Variable-Length Arguments

C's printf(const char *fmt, ...) takes a variable number of arguments. How does the calling convention handle this?

On x86-64 System V, variadic functions are called exactly like normal functions: the first 6 integer args go in registers, then the stack. The callee uses va_start, va_arg, va_end macros (defined in <stdarg.h>) to iterate over the arguments.

The tricky part: the callee doesn't know how many register arguments it received or what their types are. The ABI solves this by requiring the caller to put the number of floating-point register arguments in AL (the low byte of RAX) before calling a variadic function. This tells the callee how many XMM registers to dump onto the stack in its register-save area.

This is why you see in printf's disassembly (or any variadic function): a block that conditionally stores XMM0-XMM7 to a register save area on the stack if AL > 0. The va_arg mechanism then walks first through the register save area (for register-passed arguments) and then through the stack overflow area (for stack-passed arguments).

The key lesson: the calling convention is not just about normal functions. For variadic functions, the ABI has extra rules, and violating them (e.g., calling a variadic function through a non-variadic function pointer) is undefined behavior that can corrupt the register save area.

#include <stdarg.h>
#include <stdio.h>

// Minimal variadic sum function
long sum_n(int count, ...) {
    va_list ap;
    va_start(ap, count);  // 'count' is the last named parameter
    long total = 0;
    for (int i = 0; i < count; i++) {
        total += va_arg(ap, int);  // pop next int from arg list
    }
    va_end(ap);
    return total;
}

int main(void) {
    printf("%ld\n", sum_n(4, 10, 20, 30, 40));  // prints 100
    printf("%ld\n", sum_n(2, 7, 3));             // prints 10
    return 0;
}
// Under the hood on x86-64:
// sum_n(4, 10, 20, 30, 40) compiles to:
//   mov edi, 4    ; count in rdi
//   mov esi, 10   ; 1st vararg in rsi
//   mov edx, 20   ; 2nd vararg in rdx
//   mov ecx, 30   ; 3rd vararg in rcx
//   mov r8d, 40   ; 4th vararg in r8
//   xor eax, eax  ; 0 float args
//   call sum_n

12Kernel Conventions: asmlinkage, pt_regs, and Context Switches

The Linux kernel does not always use the standard System V ABI. It has its own calling convention requirements for specific cases:

asmlinkage: System calls enter the kernel via an assembly stub. The x86-64 kernel uses asmlinkage (a GCC attribute that forces all arguments to be passed on the stack, no registers) for syscall entry points like sys_read. Why? Because the syscall arguments are passed in registers by the userspace ABI (rdi, rsi, rdx...), but the kernel's assembly entry code saves them to the stack (into a pt_regs struct) before calling the C handler. Using stack passing ensures no register/stack mismatch.

pt_regs: When an interrupt, exception, or syscall occurs, the CPU automatically pushes a partial register state onto the kernel stack (RIP, CS, RFLAGS, RSP, SS). The kernel's entry code then pushes the remaining general-purpose registers, forming a complete struct pt_regs at the top of the kernel stack. This struct is the frame that ptrace(2), signal delivery, and strace read and write to inspect/modify user register state.

Context switch: schedule() eventually calls __switch_to_asm() in arch/x86/entry/entry_64.S. This code saves the callee-saved registers (RBX, RBP, R12-R15) and RSP of the outgoing task onto its kernel stack, then loads RSP from the incoming task's task_struct->thread.sp. When the incoming task's registers are popped (via __switch_to), it resumes exactly where it was — because what it resumes into is the stack frame it had when it was last context-switched out. The entire CPU state of a sleeping task lives on its kernel stack.

This is the payoff: stack frames are not just a language feature. They are the physical embodiment of execution state. Understanding them at this level is what makes kernel code legible.

// From arch/x86/include/asm/ptrace.h (simplified)
struct pt_regs {
    unsigned long r15, r14, r13, r12;  // pushed by entry asm
    unsigned long rbp, rbx;            // pushed by entry asm
    unsigned long r11, r10, r9, r8;
    unsigned long rax, rcx, rdx, rsi, rdi;
    unsigned long orig_rax;            // syscall number
    // --- hardware-pushed on exception/interrupt ---
    unsigned long rip;                 // instruction pointer
    unsigned long cs;                  // code segment
    unsigned long eflags;
    unsigned long rsp;                 // user stack pointer
    unsigned long ss;                  // stack segment
};
// When a syscall fires, the kernel stack looks exactly like
// a struct pt_regs at the top, followed by the kernel frames
// for do_syscall_64() -> sys_xyz() -> ...
//
// strace works by ptrace(PTRACE_GETREGS) which reads pt_regs
// directly from the tracee's kernel stack.

Key insights

  • ✦The stack grows toward lower addresses — every push decrements the stack pointer. This surprises almost everyone the first time, but it's fundamental to understanding every stack diagram you'll ever read.
  • ✦The CPU's CALL instruction only pushes the return address and jumps — everything else (saving RBP, allocating locals) is compiler-generated code in the function prologue. These are distinct mechanisms.
  • ✦Calling conventions are an ABI contract between separately compiled pieces of code. The caller-saved / callee-saved split is the core of it: whoever needs a value across a call is responsible for saving it.
  • ✦The frame pointer (RBP) is optional — modern compilers omit it for performance. But omitting it makes profiling and debugging much harder, which is why the Linux kernel is often built with it enabled.
  • ✦Recursion's real cost is stack memory, not just CPU time. Each frame lives until the function returns; N deep recursion means N frames alive simultaneously. Tail call optimization is the compiler's escape hatch.
  • ✦The red zone (128 bytes below RSP) lets leaf functions store locals without adjusting RSP at all — a neat optimization that the kernel explicitly disables because hardware interrupts would corrupt it.
  • ✦A struct pt_regs at the top of every kernel stack is not an abstraction — it is literally the user-space register file saved on entry. Context switching is just swapping stack pointers. The stack IS the execution state.

Common pitfalls

  • ⚠Confusing 'stack overflow' (stack pointer exceeds allocated stack memory, causes SIGSEGV from guard page) with 'stack buffer overflow' (writing past the end of a stack-allocated array, potentially overwriting the return address — a security vulnerability, not necessarily an immediate crash).
  • ⚠Assuming arguments are always passed in the same registers regardless of function signature. The order (RDI, RSI, RDX, RCX, R8, R9) is fixed, but floats go in XMM registers, and the 7th+ integer argument goes on the stack — forgetting this breaks hand-written assembly wrappers.
  • ⚠Writing inline assembly that clobbers caller-saved registers without listing them in the clobber list. GCC's optimizer will assume those registers are unchanged, leading to silent data corruption.
  • ⚠Relying on the frame pointer being present when it might be omitted at -O2. Writing debugger or profiler code that assumes the RBP chain exists will silently produce wrong results on optimized binaries.
  • ⚠Expecting tail call optimization to always happen. TCO is not guaranteed by the C standard — it's a compiler optimization. Code that depends on TCO for correctness (e.g., relying on it to avoid stack overflow) needs to be verified with -S output, or rewritten as an explicit loop.
  • ⚠Taking the address of a local variable and returning it from a function — the variable's stack slot is deallocated on return, giving a dangling pointer into dead stack space that will be overwritten by the next call.
  • ⚠Assuming the stack layout is fixed across compilers, compiler versions, or optimization levels. Stack offsets are an implementation detail. Code that hardcodes them (outside of carefully audited assembly) is fragile and non-portable.

Practice

  1. 01
    easy

    Write a C program that prints the exact address and value of (1) its own return address, (2) the saved RBP, and (3) each local variable, using pointer arithmetic from `__builtin_frame_address(0)`. Compile with `-O0 -fno-omit-frame-pointer` and verify the layout matches the diagram in this module.

  2. 02
    medium

    Write a function `void show_call_stack(void)` that manually walks the frame-pointer chain and prints each return address, without using `backtrace(3)` or any library function. Print at most 20 frames. Then call it from 5 levels of nested functions and verify the output matches GDB's `bt`.

  3. 03
    medium

    Use GCC's `-S` flag to compile three versions of the same function (a loop that sums an array): with `-O0`, with `-O2`, and with `-O2 -fno-omit-frame-pointer`. Write a detailed commentary annotating each assembly output: identify the prologue, epilogue, where each argument lives, which registers are caller-saved vs callee-saved, and what the compiler chose to keep in registers vs spill to the stack.

  4. 04
    hard

    Write a C program that intentionally demonstrates a stack buffer overflow writing into the return address. Use it to make the function 'return' to a different function you wrote (a simple `void win(void)` that prints 'You controlled the return address!'). This must compile and run on your machine (use `-fno-stack-protector -no-pie -O0`). Document every byte you overwrite and why.

  5. 05
    medium

    Benchmark the overhead of a frame pointer. Write a tight loop that calls a tiny function (return x+1) one billion times. Compile two versions: `-O1 -fno-omit-frame-pointer` and `-O1 -fomit-frame-pointer`. Time both with `perf stat`. Report the wall-clock difference and the instruction count difference. Explain why the difference is what it is (or is not what you expected).

Resources

📄System V Application Binary Interface — AMD64 Architecture Processor Supplement
docsfree

The authoritative ABI specification. Chapter 3 covers calling conventions, register use, and stack frame layout in exhaustive detail. Required reading for anyone writing assembly or FFI code.

⚡Compiler Explorer (godbolt.org)
interactivefree

Paste C and see the assembly output in real time, side by side. Toggle optimization flags. Compare x86-64, AArch64, and RISC-V output simultaneously. Irreplaceable for this module.

📕Computer Systems: A Programmer's Perspective (CS:APP)
bookpaidBryant & O'Hallaron, Pearson. Chapters 3 (Machine-Level Representation) covers stack frames, calling conventions, and buffer overflows at exactly the depth this module requires.

The definitive undergraduate systems textbook. Chapter 3 is 150 pages of precisely what this module covers. If you only buy one book for this roadmap, make it this one.

📕Operating Systems: Three Easy Pieces — Chapter on Address Spaces
bookfree

Free online. Provides essential context on virtual memory and how the stack region fits into a process's address space — the prerequisite picture for understanding stack growth.

📕Linux Kernel Development (3rd ed.)
bookpaidRobert Love, Addison-Wesley. Chapter on process management covers kernel stacks, context switching, and pt_regs — the kernel-side payoff of this module.

Concise and accurate. The chapters on scheduling and kernel stacks directly connect the calling-convention theory here to real kernel code.

✎x86-64 Assembly: Frame Pointer and Call Stack (LWN.net)
articlefree

LWN consistently publishes the best kernel-adjacent technical writing. Search lwn.net for 'frame pointer' and 'VMAP_STACK' to find deeply relevant articles on the kernel's stack management choices.

📄GDB Documentation — Examining the Stack
docsfree

Official GDB manual section on backtrace, frame selection, and inspecting frame internals. Reference for the GDB workflow described in this module.

📄Linux x86-64 entry code (entry_64.S)
docsfree

Read the actual kernel syscall entry assembly on Elixir (Bootlin's cross-referencer). See exactly how pt_regs is constructed and how __switch_to_asm swaps kernel stacks. Everything in this module's final concept made concrete.

Checkpoints — you can now…

  • ✓You can draw the exact byte layout of an x86-64 stack frame from memory, labeling the return address, saved RBP, callee-saved registers, and local variables with their RBP-relative offsets.
  • ✓You can compile a C function to assembly with `gcc -S`, identify the prologue and epilogue, and explain what each instruction does and why.
  • ✓You can list the six integer argument registers of the System V AMD64 ABI in order (RDI, RSI, RDX, RCX, R8, R9) and name at least three callee-saved registers.
  • ✓You can explain the difference between caller-saved and callee-saved registers, and give a concrete example of when violating this distinction would cause a bug.
  • ✓You can write a program that manually walks the frame-pointer chain to print a backtrace, and explain why it requires `-fno-omit-frame-pointer` to work correctly.
  • ✓You can explain what happens — at the level of virtual memory, CPU exception handling, and signal delivery — when a C program overflows its stack.
  • ✓You can explain what `struct pt_regs` is, why it lives at the top of the kernel stack, and how a context switch works in terms of stack pointer manipulation.