x86-64 Assembly Basics

A register is a tiny, blazingly fast storage location inside the CPU die itself. Unlike RAM, which sits on a separate chip and requires bus transactions to access, registers are part of the processor's execution units. An x86-64 core has 16 general-purpose integer registers, each 64 bits (8 bytes) wide.

The 16 general-purpose registers:

The naming looks cryptic at first. rax = register accumulator extended (the 'r' prefix and 64-bit width were added in AMD64; 'e' = 32-bit extended from 16-bit; the original 16-bit was just ax). You do not need to memorize the etymology — just know the names.

Sub-register aliasing is a key x86 quirk: writing to eax (the 32-bit view) zero-extends into rax, clearing the upper 32 bits. Writing to ax (the 16-bit view) or al (the 8-bit view) does not zero-extend — it only modifies those bits, leaving the rest intact. This is a historical quirk that the AMD64 architects deliberately chose for performance and backward compatibility. It bites people in subtle ways: if you compute a 32-bit result into eax, the upper 32 bits of rax are zero, so reading rax afterward is safe. But if you write only to al, the upper 56 bits of rax are whatever they were before.

Special-purpose registers:

• rsp (stack pointer): Always points to the most recently pushed item on the call stack. The x86-64 stack grows downward — pushing decrements rsp.
• rbp (base/frame pointer): Conventionally points to the base of the current stack frame, making it easy for debuggers to walk the call chain. Modern compilers often omit the frame pointer (-fomit-frame-pointer) as an optimization.
• rip (instruction pointer): Contains the address of the next instruction to execute. You cannot move a value into rip directly (that would be a jump), but you can read it via lea rax, [rip].
• rflags (flags register): A 64-bit register where individual bits encode CPU state after arithmetic: Zero Flag (ZF), Sign Flag (SF), Carry Flag (CF), Overflow Flag (OF), Direction Flag (DF), etc. Conditional jumps test these flags.

# See sub-register aliasing live:
# Compile with: gcc -O0 -S alias_demo.c -o alias_demo.s
# Or just read the concept:

# movl $0xDEADBEEF, %eax   -> rax = 0x00000000DEADBEEF  (zero-extended!)
# movb $0xFF, %al          -> rax = 0x00000000DEADBEFF  (only low byte changed)
# movl $1, %eax            -> rax = 0x0000000000000001  (zero-extended again)

# Demonstrate with inline asm:
#include <stdint.h>
#include <stdio.h>

int main(void) {
    uint64_t val = 0;
    __asm__ volatile (
        "movq $0xCAFEBABEDEADBEEF, %%rax\n"
        "movl $0x12345678, %%eax\n"   /* zero-extends: rax = 0x0000000012345678 */
        "movq %%rax, %0"
        : "=r"(val)
        :
        : "rax"
    );
    printf("rax = 0x%016lx\n", val);  /* prints 0x0000000012345678 */
    return 0;
}

Assembly language is a textual representation of machine instructions. The problem is that there are two mainstream syntaxes for x86 assembly, and they look significantly different. You will encounter both in the wild.

Intel syntax (used by NASM, MASM, Intel's own documentation, and MSVC):

mov rax, 42        ; destination FIRST, source second
add rax, rbx       ; rax = rax + rbx
mov rax, [rbp-8]   ; load from memory at address rbp-8

AT&T syntax (used by GAS, the GNU Assembler, and therefore gcc -S output on Linux/macOS):

movq $42, %rax     ; source FIRST, destination second
addq %rbx, %rax    ; rax = rax + rbx
movq -8(%rbp), %rax  ; load from memory at address rbp-8

The differences in a nutshell:

The operand order reversal is the most disorienting at first. AT&T reads like "move the value 42 from constant, to rax" — source-to-destination like an arrow flowing left to right. Intel reads like an assignment: rax = 42. Neither is objectively better; they are just conventions from different traditions.

The size suffixes in AT&T (q = 64-bit quad-word, l = 32-bit long, w = 16-bit word, b = 8-bit byte) look redundant when register names already encode the size (%rax is 64-bit, %eax is 32-bit), but they matter when the operand involves memory, where the size is ambiguous.

Which to learn? Learn both. The Linux kernel uses AT&T syntax (GAS). NASM/Intel syntax is more readable for learning and is used in many textbooks and CTF contexts. Compiler Explorer (godbolt.org) can show you either flavor with a toggle. Once you understand both, your brain will switch automatically.

# Same operation, both syntaxes:

# AT&T (GAS) — what gcc -S produces on Linux:
    movq    $10, %rdi        # 1st arg = 10
    movq    $20, %rsi        # 2nd arg = 20
    addq    %rsi, %rdi       # rdi = rdi + rsi = 30
    movq    %rdi, %rax       # return value
    ret

# Intel (NASM) — same logic:
    mov     rdi, 10
    mov     rsi, 20
    add     rdi, rsi         ; rdi = 30
    mov     rax, rdi
    ret

# To get AT&T from gcc:
#   gcc -O1 -S -o output.s myfile.c
# To get Intel syntax from gcc:
#   gcc -O1 -S -masm=intel -o output.s myfile.c
# To disassemble a binary in Intel syntax with objdump:
#   objdump -d -M intel ./a.out

After most arithmetic and logical instructions execute, the CPU automatically updates individual bits in the RFLAGS register to record properties of the result. These bits are called flags or condition codes. They are the CPU's way of remembering "what just happened" so that a subsequent branch instruction can act on it.

The key flags for beginners:

• ZF (Zero Flag, bit 6): Set to 1 if the result was zero. cmp a, b internally computes a - b and sets flags based on that; if a == b, the result is zero, so ZF=1.
• SF (Sign Flag, bit 7): Set to 1 if the result's most significant bit is 1 (i.e., the result is negative in two's complement).
• CF (Carry Flag, bit 0): Set to 1 if there was an unsigned carry out of (or borrow into) the most significant bit. Used for multi-precision arithmetic and unsigned comparisons.
• OF (Overflow Flag, bit 11): Set to 1 if there was a signed overflow — the result's sign is wrong for the operation performed (e.g., adding two large positive numbers yielded a negative result).
• PF (Parity Flag, bit 2): Set if the low byte of the result has an even number of 1-bits. Rarely used in modern code.
• DF (Direction Flag, bit 10): Controls the direction of string operations (movs, scas, etc.) — 0 = forward (increment), 1 = backward (decrement). Set with std, cleared with cld.

The flags are tested by conditional jump instructions like je (jump if equal / ZF=1), jne (ZF=0), jl (less than, signed: SF≠OF), jb (below, unsigned: CF=1), and so on. There are about 16 conditional jump mnemonics, but they all just test combinations of these few flags.

An important distinction: cmp and test are non-destructive flag-setters:

• cmp a, b computes a - b and sets flags, but discards the result. It is purely for comparing.
• test a, b computes a AND b and sets flags (SF, ZF, PF), discarding the result. The idiom test rax, rax is the idiomatic way to check if rax is zero (if rax AND rax == 0, then rax must be 0), and it is one byte shorter than cmp rax, 0.

# Demonstrating flags with a simple loop (AT&T syntax):
# Counts from 5 down to 0, exits when zero.

    .text
    .globl _start
_start:
    movq    $5, %rcx        # rcx = 5 (loop counter)
loop:
    decq    %rcx            # rcx -= 1; sets ZF if rcx becomes 0
    jnz     loop            # jump back if ZF == 0 (not zero yet)
    # rcx is now 0; we fall through
    movq    $60, %rax       # syscall: exit
    xorq    %rdi, %rdi      # exit code 0  (xor reg,reg is idiomatic zero)
    syscall

# The 'test' idiom for NULL check:
    movq    some_pointer(%rip), %rax
    testq   %rax, %rax      # sets ZF=1 if rax==0
    jz      handle_null     # jump if pointer is NULL

x86-64 has hundreds of instructions, but a surprisingly small set covers the vast majority of real code. Here are the essential ones with precise semantics:

Data movement:

• mov dst, src — Copy src into dst. Neither operand is modified in-place (unlike some RISC architectures). The most common instruction in any binary — often 25-40% of all instructions.
• lea dst, [expr] — Load Effective Address. Computes the address of the memory expression and stores it in dst, without actually accessing memory. Compilers love this for fast arithmetic: lea rax, [rdi + rdi*4] computes rax = rdi * 5 with a single instruction.
• xchg dst, src — Atomically swap two values. Has an implicit LOCK prefix when one operand is memory.
• push src — Decrements rsp by 8, then stores src at the new [rsp].
• pop dst — Loads the value at [rsp] into dst, then increments rsp by 8.

Arithmetic:

• add dst, src — dst += src. Sets ZF, SF, CF, OF.
• sub dst, src — dst -= src. Sets flags.
• inc dst / dec dst — Increment or decrement by 1. Note: these do not modify CF (a quirk that matters in multi-precision arithmetic). Prefer add dst, 1 when you need CF.
• neg dst — Two's complement negate: dst = 0 - dst.
• imul dst, src — Signed multiply. imul rax, rbx computes rax = rax * rbx.
• idiv src — Signed divide. Divides the 128-bit value in rdx:rax by src. Quotient → rax, remainder → rdx. You must sign-extend rax into rdx:rax using cqo before dividing.

Logical/Bitwise:

• and dst, src, or dst, src, xor dst, src, not dst — Bitwise operations. xor rax, rax is the canonical zero-a-register idiom (it is shorter and sometimes faster than mov rax, 0).
• shl dst, cl / shr dst, cl — Shift left/right (logical; fills with zeros). Count comes from cl or an immediate.
• sar dst, cl — Shift arithmetic right (fills with sign bit, preserving sign for signed integers).

Control flow:

• jmp target — Unconditional jump. Target can be a label, register, or memory.
• je/jne/jl/jle/jg/jge/jb/jbe/ja/jae — Conditional jumps (signed and unsigned variants).
• call target — Push return address (next instruction's rip) onto stack, then jump to target.
• ret — Pop return address from stack into rip; control returns to caller.
• syscall — Transitions to kernel mode to execute a system call. On Linux/x86-64: syscall number in rax, args in rdi, rsi, rdx, r10, r8, r9.

# Practical snippet: compute max(a, b) in AT&T syntax
# Arguments: rdi=a, rsi=b
# Returns: rax = max(a, b)

    .text
    .globl max_ab
max_ab:
    movq    %rdi, %rax      # rax = a (tentative result)
    cmpq    %rsi, %rdi      # compute a - b, set flags
    jge     .done           # if a >= b (signed), we already have the answer
    movq    %rsi, %rax      # else rax = b
.done:
    ret

# Compiler-idiomatic version uses cmov (conditional move):
# cmovl %rsi, %rax  — move rsi->rax if less (no branch!)
# This avoids branch misprediction on modern CPUs.

An addressing mode is the way an instruction specifies a memory location (or an immediate value or register). x86-64 has a rich set of addressing modes — richer than most modern RISC architectures — inherited from its CISC heritage.

The general form of an x86-64 memory reference is:

[base + index * scale + displacement]

where:

• base: any 64-bit register (e.g., rbp, rsp, rdi)
• index: any 64-bit register except rsp
• scale: 1, 2, 4, or 8 (powers of two; maps to byte/short/int/long-sized array element access)
• displacement: a signed 8-bit or 32-bit constant offset

Any combination of these four components can be present or absent. Examples:

The SIB byte (Scale-Index-Base) is the encoding mechanism that makes the base + index * scale + displacement form possible within a single instruction. We cover this in the encoding section.

RIP-relative addressing deserves special mention. In 64-bit mode, direct absolute addressing is rarely used. Instead, [rip + displacement] lets code reference global variables at a known offset from the instruction, without knowing the absolute address at compile time. This is what makes position-independent code (PIC) and shared libraries work: the binary can be loaded anywhere in memory, and all memory references still work because they are relative to the current instruction pointer.

The lea instruction computes these expressions without accessing memory — it is essentially a free arithmetic unit:

lea rax, [rdi + rdi*2]   ; rax = rdi * 3  (no multiply instruction needed!)
lea rax, [rsi + 7]        ; rax = rsi + 7  (pointer arithmetic)

/* C code that demonstrates several addressing modes */
#include <stdint.h>

struct Point { int64_t x, y; };

int64_t sum_array_element(int64_t *arr, int64_t i, struct Point *p) {
    /* arr[i]         -> base + index*scale addressing */
    /* p->y           -> base + displacement addressing */
    return arr[i] + p->y;
}

/* gcc -O1 -S produces approximately:
   movq (%rdi,%rsi,8), %rax    # arr[i]: rdi=arr, rsi=i, scale=8 (int64_t)
   addq 8(%rdx), %rax          # p->y: rdx=p, displacement=8 (after x)
   ret
*/

An Application Binary Interface (ABI) is a contract between separately compiled pieces of code. It answers: "where do I put arguments? where do I find the return value? which registers can I trash, and which must I preserve?"

The System V AMD64 ABI is used by Linux, macOS, and all Unix-like systems on x86-64. (Windows uses a different ABI, which is why Windows DLLs cannot be linked directly with Linux code.)

Integer/pointer argument passing (in order):

1. rdi — 1st argument
2. rsi — 2nd argument
3. rdx — 3rd argument
4. rcx — 4th argument
5. r8 — 5th argument
6. r9 — 6th argument
7. Additional arguments go on the stack, pushed right-to-left.

Return values: Integer/pointer return values ≤ 64 bits go in rax. A 128-bit return uses rdx:rax.

Caller-saved (volatile) registers: rax, rcx, rdx, rsi, rdi, r8–r11. The caller must save these before a call if it needs them after; the callee is free to clobber them.

Callee-saved (non-volatile) registers: rbx, rbp, r12–r15, rsp. If a callee wants to use these, it must save them (push onto stack) at the start and restore them (pop from stack) before returning.

Understanding the ABI is what lets you:

• Write assembly functions callable from C (and vice versa)
• Read disassembly of any C program and know what every call instruction expects
• Understand why a function that uses rbx always starts with push rbx and ends with pop rbx
• Understand the stack frame layout that GDB and debuggers display

/* C function and its faithful hand-written assembly equivalent */

/* C: */
long add3(long a, long b, long c) {
    return a + b + c;
}

/* Hand-written AT&T assembly equivalent: */
/*
    .text
    .globl add3
add3:
    # Arguments: rdi=a, rsi=b, rdx=c
    # We are allowed to clobber rdi, rsi, rdx (caller-saved)
    # Return value must go in rax
    movq    %rdi, %rax     # rax = a
    addq    %rsi, %rax     # rax += b
    addq    %rdx, %rax     # rax += c
    ret                    # return rax
*/

/* A function that must preserve rbx: */
long compute(long *arr, long n) {
    long sum = 0;
    for (long i = 0; i < n; i++) sum += arr[i];
    return sum;
}
/* The compiler will use rbx or r12-r15 for loop variables
   that survive across inner calls, and will push/pop them. */

Theory becomes real when you assemble and run actual code. This section walks through a complete, minimal Linux x86-64 program that writes "Hello, world!" to stdout and exits cleanly — with no C library. We use only Linux system calls directly.

On Linux x86-64, the syscall instruction invokes the kernel. The calling convention for Linux syscalls (not the C ABI!) is:

• rax = syscall number
• rdi, rsi, rdx, r10, r8, r9 = arguments 1–6
• After syscall, rax holds the return value (negative = error)

Key syscall numbers:

• 1 = write(fd, buf, count)
• 60 = exit(status)

The source below uses AT&T/GAS syntax because that is what the GNU toolchain (the Linux kernel's build system) uses. To assemble:

as hello.s -o hello.o
ld hello.o -o hello
./hello

Or with NASM (Intel syntax version shown in the code comment):

nasm -f elf64 hello.nasm -o hello.o
ld hello.o -o hello
./hello

Every line is explained in the code example. Pay attention to how .text and .data sections separate code from data, and how msg is referenced via %rip-relative addressing for position-independent code.

# hello.s — minimal Linux x86-64 "Hello, world!" (AT&T/GAS syntax)
# Assemble: as hello.s -o hello.o && ld hello.o -o hello && ./hello

    .section .data
msg:
    .ascii  "Hello, world!\n"   # the string (no null terminator needed)
msg_len = . - msg              # assembler arithmetic: current addr minus msg addr = 14

    .section .text
    .globl _start              # _start is the entry point; linker requires this
_start:
    # syscall: write(fd=1, buf=msg, count=msg_len)
    movq    $1,       %rax     # syscall number 1 = write
    movq    $1,       %rdi     # fd = 1 (stdout)
    leaq    msg(%rip), %rsi    # rsi = address of msg  (RIP-relative)
    movq    $msg_len, %rdx     # rdx = length
    syscall                    # enter kernel

    # syscall: exit(status=0)
    movq    $60,      %rax     # syscall number 60 = exit
    xorq    %rdi,     %rdi     # exit status = 0
    syscall

# To inspect the result:
#   objdump -d ./hello          # disassemble
#   strace ./hello              # trace system calls
#   readelf -h ./hello          # ELF header info

Every assembly instruction is ultimately a sequence of bytes. The CPU's decoder reads these bytes and figures out what operation to perform, what operands to use, and how long the instruction is. This encoding is one of the more complex aspects of x86-64 — but understanding the basic structure is enormously valuable: it explains why some instructions are 1 byte and others are 15 bytes, why disassemblers work the way they do, and how exploits like shellcode or instruction-stream confusion attacks work.

An x86-64 instruction is laid out as:

[Prefixes] [REX] [Opcode] [ModRM] [SIB] [Displacement] [Immediate]

Prefixes (0–4 bytes): Optional bytes that modify the instruction. Examples:

• 0x66 — Operand size override (16-bit instead of 32-bit)
• 0x67 — Address size override
• 0xF3, 0xF2 — REP/REPNE prefixes for string instructions
• 0xF0 — LOCK prefix (atomic operation)

REX prefix (0 or 1 byte): Introduced in AMD64. The byte has the form 0100WRXB where:

• W=1 → 64-bit operand size
• R → extends the ModRM reg field
• X → extends the SIB index field
• B → extends the ModRM r/m field or the register in the opcode byte Without REX.W, most instructions default to 32-bit operation (which zero-extends into 64-bit).

Opcode (1–3 bytes): The core instruction identifier. 0x89 is MOV r/m, r. 0x8B is MOV r, r/m. Single-byte opcodes like 0x90 (NOP) are the simplest.

ModRM byte: When present, encodes the addressing mode and two operand designators:

[ mod (2 bits) | reg (3 bits) | r/m (3 bits) ]

• mod=11: both operands are registers
• mod=00: register indirect (no displacement)
• mod=01: 8-bit displacement follows
• mod=10: 32-bit displacement follows
• When r/m=100 and mod≠11: a SIB byte follows.

SIB byte (Scale-Index-Base): Encodes [base + index * scale] when needed.

[ scale (2 bits) | index (3 bits) | base (3 bits) ]

Scale encoding: 00=1, 01=2, 10=4, 11=8.

A concrete example: movq %rbx, %rax in AT&T (move rbx into rax):

• REX.W prefix: 0x48 (0100 1000: W=1, R=0, X=0, B=0)
• Opcode: 0x89 (MOV r/m64, r64)
• ModRM: 0xC3 (11 000 011: mod=11 registers, reg=000=rax, r/m=011=rbx)
• Total: 3 bytes: 48 89 C3

You can verify this on Compiler Explorer (godbolt.org) by hovering over any instruction — it shows the encoding.

# Verifying instruction encodings with objdump and Python
# Assemble a tiny snippet:
# echo -e '.globl f\nf: movq %rbx, %rax\n ret' | as -o t.o -
# objdump -d t.o
# Output:
# 0000000000000000 <f>:
#    0: 48 89 d8    mov    %rbx,%rax
#    3: c3          ret

# Decode 48 89 d8 manually:
# 0x48 = REX prefix = 0100 1000
#        W=1 (64-bit), R=0, X=0, B=0
# 0x89 = opcode: MOV r/m64, r64
# 0xd8 = ModRM  = 1101 1000
#        mod=11 (reg-to-reg), reg=011 (rbx), r/m=000 (rax)
# Result: move rbx into rax — correct!

# Python snippet to decode a REX prefix:
def decode_rex(byte):
    assert 0x40 <= byte <= 0x4F, "Not a REX byte"
    return {
        'W': (byte >> 3) & 1,  # 64-bit operand size
        'R': (byte >> 2) & 1,  # extend ModRM.reg
        'X': (byte >> 1) & 1,  # extend SIB.index
        'B': (byte >> 0) & 1,  # extend ModRM.r/m or opcode reg
    }

print(decode_rex(0x48))  # {'W': 1, 'R': 0, 'X': 0, 'B': 0}

The fastest way to learn assembly is to generate it from C you understand, then read what the compiler produced. Two tools are indispensable:

Compiler Explorer (godbolt.org) is a browser-based tool where you type C (or C++, Rust, Go, ...) in the left pane and see the corresponding assembly in the right pane, in real time. You can:

• Choose any GCC/Clang/MSVC version and see how optimization level affects output
• Toggle between AT&T and Intel syntax
• Hover over assembly lines to see the instruction encoding
• Color-code lines to see which C source line produced which assembly
• Add multiple compilers side-by-side to compare

This is how experts read compiler output: write a tiny, focused C function, look at the asm, understand it, modify the C, and watch the asm change.

objdump is a command-line tool for dissecting compiled binaries:

objdump -d ./binary           # disassemble all code sections
objdump -d -M intel ./binary  # Intel syntax
objdump -S ./binary           # interleave source and asm (requires -g)
objdump -x ./binary           # show all section headers

readelf gives you ELF file structure:

readelf -h ./binary    # ELF header (architecture, entry point, ...)
readelf -S ./binary    # all sections (.text, .data, .rodata, ...)
readelf -l ./binary    # program (loadable) segments

strace shows what system calls your program makes:

strace ./hello         # trace every syscall with args and return values

Get comfortable with these four tools. You will use them for the rest of your systems career. A workflow: write C, compile with gcc -O1 -g, disassemble with objdump -S, and cross-reference with godbolt.org.

# Practical workflow: see what the compiler does with a loop

# 1. Write a simple function: sum.c
# long sum(long *a, long n) {
#     long s = 0;
#     for (long i = 0; i < n; i++) s += a[i];
#     return s;
# }

# 2. Compile to assembly with different optimization levels:
#    gcc -O0 -S sum.c -o sum_O0.s   # unoptimized — lots of stack spills
#    gcc -O1 -S sum.c -o sum_O1.s   # basic opts — clean register use
#    gcc -O3 -S sum.c -o sum_O3.s   # aggressive — may auto-vectorize with SSE2

# 3. Compare with objdump (after compiling to ELF):
#    gcc -O1 -g -o sum sum.c
#    objdump -d -M intel sum

# 4. On Compiler Explorer:
#    URL: https://godbolt.org/z/...  (paste your code, share link)
#    Tip: add volatile to prevent the optimizer from eliminating the loop

# 5. See the auto-vectorized version at -O3:
#    Look for vmovdqu / vpcmpeqq / vpaddq instructions — SIMD!
#    The compiler turned your scalar loop into a vector loop automatically.