Calling Conventions & ABIs (System V)

An ABI (Application Binary Interface) is a set of rules that governs how compiled binary code interacts with other compiled binary code and with the operating system. It is the binary-level analogue of an API: where an API specifies function signatures in source code, an ABI specifies the actual machine-level protocol for calling those functions.

The System V AMD64 ABI is defined in a formal document (maintained on the GitLab of the ABI working group) and covers:

• Calling conventions: which registers carry arguments, return values, and who saves what.
• Data type sizes and alignment: int is 4 bytes, long is 8 bytes on 64-bit Linux (LP64 model), structs have specific padding rules.
• Object file format: ELF (Executable and Linkable Format) sections, symbol visibility, relocation types.
• Dynamic linking: the PLT/GOT mechanism, TLS (thread-local storage) layout.
• Stack layout: how frames are organized, where the return address lives, what the red zone is.

For this module we focus entirely on the calling convention subset, since that is what you encounter in every assembly listing.

Why does it have to be written down? Because compilers are independent programs. GCC, Clang, and the Intel compiler all need to emit code that can call each other's compiled functions. The ABI is the shared contract. If GCC puts argument 1 in rdi but Clang reads argument 1 from rsi, every cross-compiled call would silently pass wrong values — a catastrophic and undetectable bug. The ABI prevents this.

# Verify the ABI on your own system:
# objdump -d shows compiled calling convention
gcc -O0 -o /tmp/demo demo.c && objdump -d /tmp/demo | grep -A20 '<main>'

The System V AMD64 ABI assigns the first six integer/pointer arguments to fixed registers, in this exact order:

Arguments 7 and beyond go on the stack, pushed right-to-left (so that argument 7 is at the lowest address, closest to the top of stack after the return address).

Floating-point arguments travel in xmm0 through xmm7 (up to 8 FP args in registers). Integer and FP argument slots are counted separately: void f(int a, double b, int c) puts a in rdi, b in xmm0, and c in rsi.

The memory trick: a mnemonic used by many — "Diane's Silk Dress Cost Real Money" — maps to rdi, rsi, rdx, rcx, r8, r9 in order. You will memorize these within a week of reading assembly regularly.

Why these registers? Historical accident plus optimization. The original i386 ABI pushed all arguments on the stack. When AMD designed the 64-bit extension, they added eight new general-purpose registers (r8–r15), and the ABI designers took the opportunity to pass arguments in registers — far faster than memory, since modern CPUs have many more physical registers than the 16 architectural ones, and register arguments can often be optimized away entirely by the compiler.

You can observe this live with Compiler Explorer (godbolt.org): write any C function with 1–8 parameters and watch which registers the compiler assigns.

// Six arguments: observe register assignments in the disassembly
// Compile: gcc -O0 -S -o demo.s demo.c

#include <stdio.h>

void six_args(long a, long b, long c, long d, long e, long f) {
    printf("%ld %ld %ld %ld %ld %ld\n", a, b, c, d, e, f);
}

int main(void) {
    six_args(1, 2, 3, 4, 5, 6);
    return 0;
}

/* Expected: the call site emits something like:
   movl $1, %edi      <- arg 1 -> rdi
   movl $2, %esi      <- arg 2 -> rsi
   movl $3, %edx      <- arg 3 -> rdx
   movl $4, %ecx      <- arg 4 -> rcx
   movl $5, %r8d      <- arg 5 -> r8
   movl $6, %r9d      <- arg 6 -> r9
   call six_args
*/

A function's return value travels back in rax (for integers and pointers up to 64 bits). For a 128-bit integer, rax holds the low 64 bits and rdx holds the high 64 bits. Floating-point return values use xmm0 (with xmm1 for a complex float pair).

Larger return types (structs, large arrays) trigger a hidden pointer mechanism: the caller allocates space for the return value (on its stack or wherever it sees fit), then passes a hidden first argument in rdi pointing to that space. The callee writes its result there and returns the same pointer in rax. This is why the calling convention effectively shifts all explicit arguments by one register when the return type is a large struct — argument 1 goes to rsi, not rdi.

Small structs (up to 16 bytes) can be returned in rax:rdx if their fields fit the integer or pointer class. The exact classification rules are detailed in the ABI document's §3.2.3 and are surprisingly nuanced — a 16-byte struct of two long values returns in rax:rdx, but a 16-byte struct containing a double and a long returns in xmm0:rax.

Practical implication for kernel work: the kernel's own syscall return convention also uses rax, but with negative values in the range [-4095, -1] representing -errno. This is not the C ABI but the kernel's internal syscall ABI; the C library (glibc/musl) translates between the two by checking if rax is in the error range and setting errno.

// Observe return value conventions
#include <stdint.h>

long return_long(void)   { return 42; }         // rax
double return_double(void) { return 3.14; }      // xmm0

typedef struct { long x; long y; } Point;
Point return_point(void) { return (Point){1, 2}; } // rax:rdx

typedef struct { long x[3]; } Big;
Big return_big(void) {                            // hidden pointer in rdi
    return (Big){{10, 20, 30}};
}

/* Disassemble with:
   gcc -O1 -S -o ret.s ret.c && cat ret.s
   Look for: movq, movaps, movq + movq (for Point), or memcpy into [rdi] (for Big) */

The ABI divides the 16 general-purpose registers into two groups based on who is responsible for preserving them across a call:

Caller-saved (a.k.a. "scratch" or "volatile") registers — the callee may freely overwrite these. If the caller needs their values after the call, the caller must save them before the call and restore them after.

Notice that all six argument registers are caller-saved. This makes sense: once you have called the function, the argument values are no longer needed by the caller (the callee has already consumed them), so there is no cost to letting the callee destroy them.

Callee-saved (a.k.a. "non-volatile" or "preserved") registers — if a callee wants to use these, it must push them on the stack at function entry and pop them before returning. The caller can assume these registers are unchanged after any call.

Also: rsp must be restored to its entry value before ret (obviously, since ret pops the return address from the stack).

Analogy: imagine a shared whiteboard in an office. Caller-saved registers are like a section of whiteboard you own during a meeting — if someone else (the callee) needs to borrow the room, they might erase your notes. You must photograph them (save to stack) before leaving. Callee-saved registers are like the permanent marker section: anyone who borrows the room must restore exactly what was there.

Why does this division exist? It is a performance optimization. If all registers were callee-saved, every function would need to push/pop all of them — massive overhead for simple leaf functions. If all were caller-saved, every call site would need to save every live register. The ABI's split is a pragmatic compromise: argument/return registers need no saving (they carry transient data), while long-lived variables kept in rbx/r12–r15 can survive calls without spilling to memory.

// Callee-saved demonstration: look at the prologue/epilogue
void uses_callee_saved(long *arr, int n) {
    long sum = 0;          // compiler will keep 'sum' in rbx or r12
    for (int i = 0; i < n; i++) {
        sum += arr[i];     // call to external func inside loop would
    }                      // force save/restore of sum's register
    arr[0] = sum;
}

/* With -O2, compiler emits approximately:
   pushq %rbx          <- save rbx (callee-saved)
   movq  $0, %rbx      <- sum = 0 in rbx
   ...loop...
   movq  %rbx, (%rdi)  <- arr[0] = sum
   popq  %rbx          <- restore rbx
   ret
*/

// Compile and check:
// gcc -O2 -S -o cs.s cs.c && grep -A30 'uses_callee' cs.s

When a function is called, the CPU executes a call instruction which does two things atomically: it pushes the return address (the address of the next instruction in the caller) onto the stack, then jumps to the callee. The stack pointer rsp thus decrements by 8 (pointing to the return address).

A typical function prologue then does:

pushq %rbp        ; save caller's frame pointer (rbp is callee-saved)
movq  %rsp, %rbp  ; establish our own frame pointer
subq  $N, %rsp    ; reserve N bytes for local variables

After the prologue, the stack looks like this (addresses decreasing downward):

  [caller's frame ...]
  arg 7 (if any)          <- rsp + 16 (relative to callee's rbp)
  return address          <- rbp + 8
  saved rbp (caller's)    <- rbp + 0
  local variable 1        <- rbp - 8
  local variable 2        <- rbp - 16
  ...                     <- rsp (bottom of frame)

The epilogue mirrors the prologue:

movq  %rbp, %rsp  ; (or 'leave' which does this + popq %rbp)
popq  %rbp        ; restore caller's frame pointer
ret               ; pop return address into rip

Frame pointer omission (-fomit-frame-pointer): modern compilers often omit rbp as a frame pointer entirely, freeing it as a general-purpose register. They instead use DWARF .cfi_* annotations in the object file to describe how to unwind the stack without rbp. The kernel compiles with -fno-omit-frame-pointer by default for reliable stack unwinding in perf, ftrace, and crash dumps — a deliberate ABI-adjacent policy decision.

Stack growth direction: on x86-64 (and nearly all architectures), the stack grows downward — toward lower addresses. rsp decrements to allocate, increments to free. This confuses beginners who think of stacks as growing up. Push is subtract-and-store; pop is load-and-add.

// Observe a real stack frame
// Compile: gcc -O0 -fno-omit-frame-pointer -S -o frame.s frame.c

long add(long a, long b) {
    long result = a + b;   // stored as local variable
    return result;
}

/* Expected output (AT&T syntax):
add:
    pushq   %rbp
    movq    %rsp, %rbp
    movq    %rdi, -24(%rbp)   ; spill arg a
    movq    %rsi, -32(%rbp)   ; spill arg b
    movq    -24(%rbp), %rdx
    movq    -32(%rbp), %rax
    addq    %rdx, %rax
    movq    %rax, -8(%rbp)    ; result = a+b
    movq    -8(%rbp), %rax    ; return value in rax
    popq    %rbp
    ret
*/

The System V AMD64 ABI requires that rsp is 16-byte aligned at the point of a call instruction. This means that on entry to any function (after the call has pushed the 8-byte return address), rsp is 16-byte aligned minus 8, i.e., rsp % 16 == 8.

Why 16 bytes? Because SSE and AVX instructions (like movaps, movdqa, vmovaps) operate on 128-bit or 256-bit values and require their memory operands to be naturally aligned. If you try to movaps [rsp], xmm0 with rsp % 16 != 0, you get a general protection fault — a crash that looks inexplicable unless you know this rule.

The convention in practice: before any call, you must ensure rsp % 16 == 0. Since call subtracts 8 (for the return address), rsp will be 16-byte aligned on entry to the callee (i.e., rsp % 16 == 8 after the push). The callee's pushq %rbp then restores alignment to 16. When the callee allocates local variables with subq $N, %rsp, it typically rounds N up to the nearest multiple of 16.

Common alignment bugs in hand-written assembly: if you push an odd number of 8-byte values before a call, you break alignment. The fix is to either push a dummy value (pushq $0) or subtract an extra 8 from rsp manually.

Verification: you can observe this with gdb: set a breakpoint at any function entry and print rsp % 16 — you should always see 8 (because the call already pushed the return address).

; Hand-written assembly: correct alignment before call
; File: align_demo.asm  (NASM syntax)
section .text
global _start
extern printf

_start:
    ; Before this call, rsp is 16-byte aligned (OS guarantees it).
    ; 'call' will push 8 bytes -> rsp % 16 == 8 inside the callee.
    ; We need rsp % 16 == 0 RIGHT BEFORE the call instruction.
    ; Currently rsp % 16 == 0, so no adjustment needed.

    lea  rdi, [rel fmt]    ; arg 1: format string
    mov  rsi, 42           ; arg 2: value
    xor  eax, eax          ; al = 0: no xmm args
    call printf            ; rsp goes from 0-aligned to 8-aligned inside printf

    ; Exit
    mov  eax, 60
    xor  edi, edi
    syscall

section .data
fmt: db "Value: %ld", 10, 0

; Build: nasm -f elf64 align_demo.asm && ld -o align_demo align_demo.o -lc --dynamic-linker /lib64/ld-linux-x86-64.so.2

The System V AMD64 ABI defines a red zone: a 128-byte area below rsp that a leaf function (one that does not call other functions) may use as scratch space without adjusting rsp. This space is guaranteed not to be clobbered by signal handlers or asynchronous interrupts — the OS and hardware context-switching machinery will not touch the 128 bytes below rsp of a user-space function.

Why does this exist? Performance. Every subq $N, %rsp and matching addq $N, %rsp in a function prologue/epilogue costs cycles. For tiny leaf functions (the most common case), the red zone eliminates this overhead entirely: the compiler can just use [rsp-8], [rsp-16], etc., directly.

What compilers do with it: GCC and Clang aggressively use the red zone in leaf functions when optimization is enabled. You will see code like:

mov  %rdi, -8(%rsp)    ; store arg 1 in red zone (no pushq needed)
mov  %rsi, -16(%rsp)   ; store arg 2 in red zone

...without any sub $N, %rsp at all.

Critical exception — the Linux kernel: the kernel compiles with -mno-red-zone. Why? Because in kernel mode, NMIs (Non-Maskable Interrupts) and MCEs (Machine Check Exceptions) can arrive at any point and use the same stack as the interrupted code. If kernel code were using the red zone and an NMI fired, the NMI handler's stack would overwrite the very memory the interrupted function was using. The kernel uses IST (Interrupt Stack Table) entries for some vectors, but the conservative and correct solution is simply to never use the red zone in kernel code.

This is a beautiful example of a point where the kernel deliberately diverges from the user-space ABI — not arbitrarily, but for a precise and defensible reason.

// Observe red zone usage: compile as leaf function with -O2
// gcc -O2 -S -o rz.s rz.c

long compute(long a, long b) {
    // No calls made, so compiler uses red zone for temporaries
    long t1 = a * b;
    long t2 = t1 + a;
    return t2 - b;
}

/* With -O2 this will likely be register-only (no stack use at all).
   Force red zone use with more temporaries and -O0:
   gcc -O0 -S -o rz0.s rz.c
   You'll see negative rsp offsets WITHOUT a 'sub $N, %rsp' prologue.
   That's the red zone. */

// Kernel comparison: with -mno-red-zone, the compiler MUST use
// sub $N, %rsp before any local variable access.
// Check: grep -r 'mno-red-zone' /path/to/linux/arch/x86/Makefile

Variadic functions like printf(const char *fmt, ...) accept a variable number of arguments. The ABI handles this with a specific rule: rax (specifically al, the low byte) must be set to the number of floating-point arguments passed in xmm registers before calling a variadic function. This allows the callee to skip scanning XMM registers it knows were not used.

The variadic mechanism in C uses the va_list type, which on AMD64 is a struct containing:

• gp_offset: offset into a saved-register area for the next integer register argument.
• fp_offset: offset for the next FP register argument.
• overflow_arg_area: pointer to stack arguments (argument 7+).
• reg_save_area: pointer to a save area where all six integer and eight FP registers were dumped on entry.

When printf calls va_arg(ap, int), it increments gp_offset and reads from reg_save_area. When all register slots are consumed, it reads from overflow_arg_area. This is why the prologue of a variadic function looks unusually heavy — it must save all argument registers to the reg_save_area immediately, since it does not know which ones were actually passed.

Practical significance: when writing assembly that calls a variadic function, you must zero rax (or set it to the xmm argument count). Forgetting this is a classic bug: the callee may try to read XMM registers that were never set, causing garbage output or crashes.

; Calling printf from assembly (correct)
; File: variadic_call.asm (NASM syntax)
section .text
global _start
extern printf

_start:
    sub  rsp, 8          ; align stack (rsp was 16-byte aligned; sub 8 -> 8-aligned; call pushes 8 -> 16-aligned in printf)
    lea  rdi, [rel fmt]  ; arg 1: format string
    mov  rsi, 100        ; arg 2: integer
    xor  eax, eax        ; CRITICAL: al=0, no xmm args
    call printf
    add  rsp, 8

    mov  eax, 60
    xor  edi, edi
    syscall

section .data
fmt: db "Result: %d", 10, 0

; If you forget 'xor eax, eax', printf may look at garbage in xmm0-xmm7

Passing structs is where the ABI gets genuinely intricate. The ABI classifies each struct field into one of several classes: INTEGER (fits in a GP register), SSE (floating point), SSEUP (upper half of a 256-bit value), X87 (80-bit floats), MEMORY, and a few others.

The classification algorithm works on 8-byte chunks (called eightbytes) of the struct:

1. If the total struct size exceeds 16 bytes, or if any field requires alignment greater than 16 bytes, the whole struct is passed MEMORY — via the hidden-pointer mechanism described in the return-value section.
2. Otherwise, each 8-byte chunk gets the "union" of its fields' classes (INTEGER wins over SSE, MEMORY wins over all).
3. If both eightbytes are classified as INTEGER, they go in two GP registers (consuming two argument register slots).
4. If the first is INTEGER and second is SSE, first goes in a GP register and second in an XMM register.

Example: struct { int x; float y; } (8 bytes total, one eightbyte): x is INTEGER class, y is SSE class — INTEGER wins, so the whole struct is passed in a single GP register with the int in the low 32 bits and the float bits in the high 32 bits.

Why this matters for kernel code: the kernel passes most complex data via pointers, avoiding struct-by-value passing entirely. But understanding the classification rules is essential when writing kernel code that interacts with user-space via ptrace, seccomp, or when you are reading compiler output to understand a performance regression.

// Struct passing classification demonstration
// Compile with: gcc -O1 -S -o struct_pass.s struct_pass.c

// Small struct: fits in one register
typedef struct { int a; int b; } TwoInts;   // passed in rdi (both ints packed)

// 16-byte struct: two register slots
typedef struct { long x; long y; } TwoLongs; // rdi=x, rsi=y

// Over 16 bytes: passed by hidden pointer
typedef struct { long x; long y; long z; } ThreeLongs; // rdi = hidden ptr

void accept_two_ints(TwoInts s)   { (void)s; }
void accept_two_longs(TwoLongs s) { (void)s; }
void accept_three_longs(ThreeLongs s) { (void)s; }

void caller(void) {
    accept_two_ints((TwoInts){1, 2});
    accept_two_longs((TwoLongs){10, 20});
    accept_three_longs((ThreeLongs){100, 200, 300});
}

/* Observe in the disassembly:
   - TwoInts: movl $1, edi; movk $2<<32...; one call
   - TwoLongs: movl $10, edi; movl $20, esi; one call
   - ThreeLongs: sub rsp for temp struct, lea rdi,[rsp+...], fill, call */

The system call ABI is a closely related but distinct convention from the C function-call ABI. When user-space code invokes a Linux system call via the syscall instruction:

The critical difference from the C ABI: argument 4 is r10, not rcx. This is because the syscall instruction itself destroys rcx (it saves the return address there so it can restore rip on sysret) and also destroys r11 (saves rflags there). The kernel therefore cannot receive argument 4 in rcx — it is already overwritten by the time the kernel entry code runs.

The kernel's entry code in arch/x86/entry/entry_64.S saves all user registers to a pt_regs struct on the kernel stack, then dispatches to the C system call handler. The C handler receives its arguments in rdi, rsi, rdx, r10, r8, r9 (via a macro that copies r10 to rcx to match the C ABI for the kernel-side function). This elegant adaptation is a miniature example of ABI translation.

Return convention: a successful syscall returns a non-negative value in rax. Errors return a negative errno value (e.g., -ENOENT is -2). The C library wrapper checks this and converts it to the user-space convention of -1 + setting errno.

; Direct system call: write(1, msg, 13) then exit(0)
; No C library involved — raw ABI
section .text
global _start

_start:
    ; write(fd=1, buf=msg, count=13)
    mov  rax, 1          ; syscall number: write
    mov  rdi, 1          ; arg1: fd = stdout
    lea  rsi, [rel msg]  ; arg2: buf
    mov  rdx, 13         ; arg3: count
    syscall              ; rcx and r11 are destroyed by this instruction
    ; rax now holds bytes written (or -errno on error)

    ; exit(0)
    mov  rax, 60         ; syscall number: exit
    xor  rdi, rdi        ; arg1: exit code = 0
    syscall

section .data
msg: db "Hello, world", 10

; Build: nasm -f elf64 syscall_demo.asm -o syscall_demo.o && ld -o syscall_demo syscall_demo.o
; Run:   ./syscall_demo

Theory becomes intuition only when you can look at real disassembly and narrate what every instruction is doing in terms of the ABI. Let us walk through a non-trivial example.

Consider this C function:

long dot_product(const long *a, const long *b, int n) {
    long sum = 0;
    for (int i = 0; i < n; i++) sum += a[i] * b[i];
    return sum;
}

When compiled with -O2, the compiler might emit:

dot_product:
    xor   eax, eax        ; sum = 0 (zeroing eax also zeroes rax)
    test  edx, edx        ; n == 0?
    jle   .done           ; if n <= 0, return 0
    xor   ecx, ecx        ; i = 0
.loop:
    movsx r8, dword [rdi + rcx*8]   ; wait — actually arrays of long:
    mov   r8, [rdi + rcx*8]         ; a[i]
    mov   r9, [rsi + rcx*8]         ; b[i]
    imul  r8, r9                    ; a[i] * b[i]
    add   rax, r8                   ; sum += product
    inc   ecx                       ; i++
    cmp   ecx, edx                  ; i < n?
    jl    .loop
.done:
    ret

ABI narration:

• rdi = pointer a (arg 1), rsi = pointer b (arg 2), rdx/edx = n (arg 3, int, sign-extended). Matches the register table exactly.
• rax is the return value register; xor eax, eax sets it to 0 efficiently.
• rcx, r8, r9 are all caller-saved — the compiler freely uses them as temporaries with no save/restore.
• No pushq %rbp, no subq $N, %rsp — this is a leaf function (no calls), and the compiler may use the red zone or pure registers.
• The ret at the end pops the return address pushed by call into rip.

This narration maps every instruction to an ABI rule. After enough practice, it becomes automatic.

// Exercise: compile and annotate
// gcc -O2 -S -o dot.s dot.c && cat dot.s
// Then annotate each line with: 'ABI reason'

#include <stddef.h>

long dot_product(const long *a, const long *b, int n) {
    long sum = 0;
    for (int i = 0; i < n; i++)
        sum += a[i] * b[i];
    return sum;
}

int main(void) {
    long a[] = {1, 2, 3};
    long b[] = {4, 5, 6};
    long result = dot_product(a, b, 3);
    return (int)result; // should be 32 (1*4+2*5+3*6)
}