System Calls & the User/Kernel Boundary

Before a single syscall can happen, you need to understand why the boundary exists at all — and the answer is in the CPU hardware itself.

Modern CPUs implement privilege levels (Intel calls them rings; ARM calls them exception levels; RISC-V calls them privilege modes). The idea is simple and powerful: the CPU tracks which privilege level the currently-executing code is at, and enforces different rules for each level.

x86-64 rings:

• Ring 0 (kernel mode): Full access. Code here can execute any instruction — including ones that halt the CPU, disable interrupts, reprogram the memory management unit (MMU), or read/write any physical address.
• Ring 3 (user mode): Restricted access. Code here cannot execute privileged instructions. Attempting to do so triggers a CPU exception. Rings 1 and 2 exist but are unused by Linux.

ARM Exception Levels (AArch64):

• EL0: User mode (applications)
• EL1: OS kernel
• EL2: Hypervisor
• EL3: Secure monitor (firmware)

RISC-V modes: User (U), Supervisor (S), Machine (M) — Linux runs the kernel in S-mode, user programs in U-mode.

The critical insight: the privilege level is stored in a CPU register (the CS register's RPL field on x86, SPSR_EL1 on ARM, mstatus.MPP on RISC-V). The CPU checks this register on every single memory access and privileged instruction. You cannot fake it from user space — only a hardware-defined transition mechanism (the trap/syscall instruction) can change it.

Why does this matter? Without this, any program could scribble on any memory address, disable interrupts to freeze the machine, or reprogram the disk controller directly. The OS's entire security model rests on this hardware primitive. The kernel is just software, but it is software that runs at a privilege level you cannot reach without asking.

# See the privilege level your process runs at:
# On Linux x86-64, try reading CR0 (a privileged register)
# from user space — it will SIGSEGV (general protection fault)

# In gdb, break anywhere in a user process:
# (gdb) info registers cs
# cs  0x33  51
# The value 0x33 = ring 3 (user). Kernel code shows 0x10 = ring 0.

A system call is a request from a user-space program to the kernel to perform a privileged action on its behalf. It is the only sanctioned way to cross from ring 3 into ring 0.

Think of it like this: you are a visitor in a secure government building (user space). You cannot walk into the classified archives yourself. But there is a service window (the syscall interface). You fill out a form (put a syscall number and arguments in specific registers), hand it through the window (execute the syscall instruction), and a government employee (the kernel) goes into the archives on your behalf, comes back, and hands you the result.

Every interaction between user-space and the OS kernel goes through this window. When you:

• Open a file → open() or openat() syscall
• Read from a socket → read() or recvfrom() syscall
• Allocate memory beyond your current break → brk() or mmap() syscall
• Create a new process → fork() / clone() syscall
• Exit → exit_group() syscall

Linux on x86-64 has about 340 syscalls (the list grows slowly). Each has a number — a small integer. read is 0, write is 1, open is 2, stat is 4. These numbers are stable ABI — the kernel promises they will never change for a given architecture. That stability is why binaries compiled 20 years ago still run on modern Linux.

Important: a library function like printf or fwrite is not a syscall. It is C code running in user space that may or may not eventually call a syscall (write in this case). The distinction matters enormously for performance and for understanding what can fail.

// Calling write() two ways:
// 1. Via the C library wrapper (libc)
#include <unistd.h>
write(1, "hello\n", 6);  // libc wrapper around the syscall

// 2. Direct syscall via Linux's syscall() helper
#include <sys/syscall.h>
syscall(SYS_write, 1, "hello\n", 6);  // same result

// 3. Raw assembly (x86-64): syscall number in rax, args in rdi,rsi,rdx
// mov rax, 1        ; SYS_write = 1
// mov rdi, 1        ; fd = stdout
// lea rsi, [msg]    ; buf
// mov rdx, 6        ; count
// syscall           ; trap into kernel

The actual crossing uses a special CPU instruction that atomically changes the privilege level and jumps to a kernel-specified address. On modern x86-64 this instruction is syscall (introduced with AMD64; older x86 used int 0x80 or sysenter). On AArch64 it is svc #0. On RISC-V it is ecall.

What syscall does on x86-64 (atomically, in hardware):

1. Saves the user-space instruction pointer (RIP) into RCX.
2. Saves the user-space flags (RFLAGS) into R11.
3. Loads the new RIP from the LSTAR MSR (a register the kernel set during boot to point at entry_SYSCALL_64 in the kernel).
4. Switches CS and SS to kernel segment selectors (ring 0).
5. Does not switch the stack — that happens in software (the kernel's entry code does it immediately).

The kernel's entry_SYSCALL_64 (in arch/x86/entry/entry_64.S) then:

1. Saves all user-space registers onto the kernel stack.
2. Switches to the process's kernel stack.
3. Looks up the syscall number (in RAX) in the syscall table.
4. Calls the corresponding kernel function.
5. On return, restores registers and executes sysret to return to ring 3.

Why can't user space fake this? The LSTAR MSR is writable only from ring 0. So the kernel, during boot, writes the address of its own entry point there. When syscall fires, the CPU always jumps to that address — you cannot redirect it. The kernel is always in control of what code runs at ring 0.

The interrupt/exception parallel: Before syscall/sysenter, the x86 int 0x80 instruction triggered a software interrupt. The CPU looked up gate descriptor 0x80 in the IDT (Interrupt Descriptor Table), which also pointed to a kernel-controlled handler. The principle is the same; syscall is just faster because it skips the IDT lookup.

# Disassemble a syscall in a real binary:
$ objdump -d /usr/lib/x86_64-linux-gnu/libc.so.6 | grep -A5 'syscall'
# You'll see the pattern:
#   mov    eax,<syscall_number>
#   syscall
#   ret

# On AArch64:
$ objdump -d /lib/aarch64-linux-gnu/libc.so.6 | grep -A3 'svc'
# You'll see:
#   mov    x8, <syscall_number>   (syscall nr goes in x8 on ARM64)
#   svc    #0x0

Once the kernel's entry code fires, it needs to know which syscall to run. The mechanism is the syscall table: a simple array of function pointers, indexed by the syscall number.

On x86-64, this table is defined in arch/x86/entry/syscall_64.c:

asmlinkage const sys_call_ptr_t sys_call_table[] = {
    [0]  = sys_read,
    [1]  = sys_write,
    [2]  = sys_open,   // actually sys_openat on newer kernels
    ...
    [62] = sys_kill,
    ...
};

The generated table comes from a script that reads arch/x86/entry/syscalls/syscall_64.tbl. Each line in that file maps a number to a name to a kernel function. For example:

0   common  read        sys_read
1   common  write       sys_write
2   common  open        sys_open

How dispatch works in entry_SYSCALL_64:

1. The syscall number is in RAX.
2. The kernel checks it against NR_syscalls (the table length).
3. It indexes sys_call_table[rax] and calls the function.
4. The return value (a long) goes into RAX before sysret.

Architecture independence: Each architecture (x86, ARM, RISC-V, MIPS, ...) has its own syscall table and its own numbering. That is why SYS_write is 1 on x86-64, 4 on x86 (32-bit), and 64 on AArch64. The <sys/syscall.h> header provides the right number for your arch. The kernel maintains separate entry paths and tables per architecture, but the actual syscall implementation (SYSCALL_DEFINE3(write, ...)) is shared across architectures in fs/read_write.c.

Security layer: Before reaching the table, the entry path checks seccomp filters (if enabled). Seccomp can deny or redirect syscalls based on a BPF program — this is how containers and sandboxes (Chrome, Docker, systemd) restrict what syscalls a process can make.

// In the kernel source: how a syscall is defined
// File: fs/read_write.c

SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
                size_t, count)
{
    return ksys_write(fd, buf, count);
}

// SYSCALL_DEFINE3 is a macro that:
// 1. Creates sys_write() with the right signature
// 2. Handles 32-bit compat wrappers
// 3. Hooks into tracing/ftrace infrastructure
// The __user annotation tells the kernel (and sparse, a static checker)
// that 'buf' is a user-space pointer — it must be copied with
// copy_from_user(), not dereferenced directly.

How do arguments travel from your user-space call into the kernel? Not on the stack (the kernel stack is separate!) — through CPU registers, following an architecture-specific calling convention.

x86-64 syscall argument registers:

Linux syscalls take at most 6 arguments. If you need more, a struct pointer is passed (e.g., clone3 takes a pointer to clone_args).

AArch64: Syscall number in X8, arguments in X0–X5. RISC-V: Syscall number in A7, arguments in A0–A5.

The pointer problem — copy_from_user: When an argument is a pointer (like the buffer in write), the kernel faces a security challenge: the user gave us a virtual address from their address space. That address:

1. Might be null or point to kernel memory (a privilege-escalation attempt).
2. Might be valid now but become invalid (unmapped) while the kernel is reading it.
3. On x86 with SMAP (Supervisor Mode Access Prevention), the CPU will fault if the kernel directly dereferences a user-space pointer.

The solution: copy_from_user(kernel_buf, user_ptr, size) and copy_to_user(user_ptr, kernel_buf, size). These functions:

1. Validate that the user address range is actually in user space.
2. Use special instructions (STAC/CLAC on x86 to temporarily disable SMAP, or use __uaccess_begin/__uaccess_end).
3. Handle page faults gracefully (via exception table entries) if the user page is not present.

This is one of the most security-critical kernel APIs. Forgetting it or using it wrong is a classic kernel vulnerability class.

// Minimal correct kernel write implementation sketch
SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
                size_t, count)
{
    char kbuf[4096];
    size_t to_copy = min(count, sizeof(kbuf));

    // SAFE: validate + copy from user space
    if (copy_from_user(kbuf, buf, to_copy))
        return -EFAULT;  // user pointer was bad

    // Now kbuf is in kernel space, safe to use
    // ... write kbuf to fd ...
    return to_copy;
}

// WRONG (never do this in real kernel code):
// char c = *buf;  // Direct dereference of user pointer!
//                 // SMAP will fault; security hole if SMAP absent

This confusion trips up nearly every new systems programmer. Let us be precise.

A library function (e.g., fopen, printf, malloc) is C code compiled into a shared library (libc — GNU libc, musl, etc.) that runs entirely in user space at ring 3. It may or may not call a syscall.

A syscall wrapper (e.g., libc's open(), read(), write()) is a thin function in libc whose sole job is to put the right number in RAX, put the arguments in the right registers, execute syscall, and return the result. The wrapper adds essentially zero logic.

Examples of the relationship:

• printf(fmt, ...) → formats the string in user space → calls write(1, buf, len) (the wrapper) → executes syscall with rax=1.
• malloc(n) → manages a heap in user space → occasionally calls brk() or mmap() to grow the heap; most malloc calls never touch the kernel at all.
• fopen(path, mode) → calls open() wrapper → syscall openat(AT_FDCWD, path, flags, mode).
• strlen(s) → pure computation, never a syscall.

Why layer it? The kernel's syscall interface is minimal and POSIX-ish but not always convenient. Libc adds:

• Buffered I/O (stdio): instead of one write syscall per putchar, buffer 4 KB and flush at once.
• Error translation: the kernel returns -ENOENT; libc sets errno = ENOENT and returns -1.
• Platform portability: your code says open(), libc decides whether to use open or openat.
• Signal restart (EINTR handling).

Bypassing libc entirely: It is valid (and instructive) to write a program that makes syscalls with no libc at all. Toy OS loaders, busybox, and musl do this. You can do it in C with syscall(), or in assembly directly.

// A complete "hello world" using ONLY direct syscalls, no libc
// Compile: gcc -nostdlib -static -o hello hello.c

#include <sys/syscall.h>

void _start(void) {  // no main() — this IS the entry point
    const char msg[] = "hello, kernel\n";
    // SYS_write = 1 on x86-64
    syscall(SYS_write, 1, msg, sizeof(msg) - 1);
    // SYS_exit_group = 231
    syscall(SYS_exit_group, 0);
    __builtin_unreachable();
}

Crossing the user/kernel boundary is not free. Understanding the cost explains a lot of Linux performance lore.

Direct hardware costs of syscall/sysret:

• Saving/restoring registers (many GPRs go to/from the kernel stack).
• The syscall instruction itself: ~100–300 ns on modern hardware (vs ~1 ns for a regular function call).

Indirect costs — the hidden majority:

1. Stack switch: The CPU switches from the user stack to the per-process kernel stack. This is a memory access that may miss L1/L2 cache.
2. TLB and cache effects: The kernel's code and data are not in the CPU's caches before the syscall. Warming them up takes time. (This is less of an issue for hot syscalls that happen constantly.)
3. Spectre/Meltdown mitigations (the big one post-2018): Meltdown required a mitigation called KPTI (Kernel Page Table Isolation). The kernel now maintains separate page tables for user and kernel mode. On each ring crossing, the CPU must switch CR3 (the page table base register) and flush most of the TLB. This alone adds ~100–1000 ns depending on the workload. On CPUs with PCID (Process-Context Identifiers), the TLB flush is partial and cheaper.
4. Retpoline / IBRS for Spectre: Indirect branch prediction must be poisoned or avoided on speculative paths, adding a few more cycles.

Practical magnitude:

• A fast syscall like getpid() (no KPTI-exempt): ~50–200 ns.
• A read() on a pipe: ~300–500 ns for the crossing alone, plus the data copy.
• An open() on a cold file: microseconds to milliseconds, dominated by VFS and filesystem logic, not the crossing.

What this drives:

• Batch syscalls: readv/writev, sendmmsg, recvmmsg — do more per crossing.
• io_uring: Submit many I/O operations in shared memory, kernel drains the ring; can achieve zero-syscall steady state.
• vDSO: Map kernel code into user space for syscalls that don't need protection (see next concept).
• Buffered I/O: stdio's whole purpose is to reduce write syscalls.

// Measure syscall overhead with clock_gettime:
#include <time.h>
#include <stdio.h>

int main(void) {
    struct timespec t1, t2;
    const int N = 1000000;

    clock_gettime(CLOCK_MONOTONIC, &t1);
    for (int i = 0; i < N; i++)
        clock_gettime(CLOCK_MONOTONIC, &t2);  // may use vDSO
    clock_gettime(CLOCK_MONOTONIC, &t2);

    long ns = (t2.tv_sec - t1.tv_sec) * 1000000000L
              + (t2.tv_nsec - t1.tv_nsec);
    printf("%.1f ns per call\n", (double)ns / N);
    // Compare with: strace -c ./a.out to see real syscall counts
    return 0;
}

The vDSO (virtual Dynamic Shared Object) is one of the most elegant performance hacks in the Linux kernel. It solves a specific problem: some syscalls are called extremely frequently, never modify system state, and only need to read a value that the kernel keeps up-to-date anyway. Why pay the trap overhead?

How it works:

1. The kernel maps a small shared memory region into every process's address space (automatically, no dlopen needed). You can see it in /proc/self/maps as [vdso].
2. This region contains actual executable kernel code — real functions compiled from arch/x86/vdso/ — but they run at ring 3.
3. The kernel also maps a read-only data page (the vvar page) that the kernel updates at high frequency (every timer tick) and that the vDSO code reads directly.
4. When libc calls gettimeofday(), it calls the vDSO version, which reads the time from the shared vvar page using a seqlock for consistency — no trap, no ring switch, just a memory read.

Which syscalls use vDSO on x86-64?

• gettimeofday() — reads wall clock
• clock_gettime() — reads monotonic/realtime clocks
• time() — coarse wall clock
• getcpu() — which CPU/NUMA node you're on

Performance impact: clock_gettime(CLOCK_REALTIME, ...) via vDSO: ~5–20 ns. Via actual syscall: ~100–500 ns. A 10–50× speedup for code that timestamps every event.

The seqlock for consistency: The kernel holds a sequence counter. Before updating the time data, it increments the counter (making it odd = write in progress). After updating, it increments again (even = stable). The reader checks: read counter, read data, read counter again. If both reads give the same even value, the data is consistent. Otherwise, retry. This is a lock-free, wait-free read in the common case.

Why not vDSO everything? Only safe for read-only, stateless queries. write(), open(), or anything that modifies kernel state must trap — there is no way to modify kernel data structures from ring 3.

// See the vDSO in your process:
#include <stdio.h>

int main(void) {
    FILE *maps = fopen("/proc/self/maps", "r");
    char line[256];
    while (fgets(line, sizeof(line), maps))
        if (strstr(line, "vdso") || strstr(line, "vvar"))
            printf("%s", line);  // e.g.: 7ffe3d9fe000-7ffe3da00000 r-xp [vdso]
    fclose(maps);
    return 0;
}

// Confirm clock_gettime uses vDSO (no syscall appears in strace):
// $ strace -e trace=clock_gettime ./a.out
// (should show 0 or very few clock_gettime syscalls)

You do not need to read kernel source to see syscalls — Linux gives you powerful tools to observe them live.

strace attaches to a process using ptrace() (itself a syscall) and intercepts every syscall entry and exit. It prints the syscall name, arguments (decoded into human-readable form), return value, and timing. This is invaluable for debugging mystery failures, understanding what a binary does, and learning what syscalls a program actually uses.

strace ls /tmp          # trace ls, print all syscalls
strace -c ls /tmp       # summary: count + time per syscall
strace -e read,write ls # only trace read and write
strace -p <pid>         # attach to a running process

How strace works internally: It uses PTRACE_SYSCALL — each syscall entry and exit causes a SIGSTOP, strace wakes up, reads the registers via ptrace(PTRACE_GETREGS), logs them, and lets the process continue. This is why strace can slow down a program by 10–100×; the overhead is the extra ptrace trap per syscall.

perf stat gives you aggregate counts without the per-syscall overhead:

perf stat -e 'syscalls:sys_enter_*' ./myprogram   # count by syscall type

/proc/self/syscall: At any moment, read /proc/<pid>/syscall to see which syscall the process is currently blocked in (or not a syscall if it's running). Useful for debugging stuck processes.

Kernel tracing with ftrace: echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable turns on kernel-side syscall tracing for all processes — useful for system-wide analysis. bpftrace and BPF-based tools give even more power with minimal overhead.

# Real strace session: what does 'cat /etc/hostname' do?
$ strace cat /etc/hostname 2>&1 | grep -E 'open|read|write'
# openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
# read(3, "mybox\n", 131072)          = 6
# write(1, "mybox\n", 6)              = 6

# Count syscalls for a build:
$ strace -c make -j4 2>&1 | tail -20
# % time     seconds  usecs/call     calls    errors syscall
# ...
# 45.12    0.123456          12     10000           stat
# 30.01    0.082000          16      5000           open

Syscalls can fail. The kernel and libc have a clean convention for reporting errors that every systems programmer must know cold.

Kernel side: A syscall returns a long. On success, this is the return value (e.g., the number of bytes read, or the file descriptor). On failure, the kernel returns a negative errno value — e.g., -ENOENT (no such file, value -2), -EACCES (permission denied, -13), -EAGAIN (-11). The values are defined in include/uapi/asm-generic/errno-base.h and errno.h.

Libc side: The libc wrapper checks if the return value is in the range [-4096, -1] (the Linux errno range). If so, it:

1. Sets the thread-local variable errno to the absolute value (ENOENT = 2).
2. Returns -1 to the caller.

So from C: if (open(path, O_RDONLY) == -1) { perror("open"); } — errno is set and perror prints the human-readable message.

Why thread-local? errno is declared as __thread int errno (or _Thread_local). Each thread has its own errno, so a syscall failure on thread A does not clobber thread B's errno.

EINTR — the special one: If a syscall is interrupted by a signal (e.g., SIGCHLD arrives while read() is blocked), the kernel returns -EINTR. Libc usually restarts the call automatically if SA_RESTART is set on the signal handler, but not always. Robust code handles EINTR manually:

retry:
    n = read(fd, buf, count);
    if (n == -1 && errno == EINTR) goto retry;

The error values are ABI: like syscall numbers, errno values are frozen. ENOENT is 2 on Linux x86 forever. Changing it would break every binary.

// Robust read() with EINTR handling
#include <unistd.h>
#include <errno.h>

ssize_t read_all(int fd, void *buf, size_t n) {
    char *p = buf;
    size_t remaining = n;
    while (remaining > 0) {
        ssize_t got = read(fd, p, remaining);
        if (got == -1) {
            if (errno == EINTR) continue;   // interrupted, retry
            return -1;                       // real error
        }
        if (got == 0) break;                 // EOF
        p += got;
        remaining -= got;
    }
    return n - remaining;
}

Adding a syscall to the kernel is intentionally hard — it is an ABI commitment that lasts decades. The kernel community's bar is extremely high. But understanding the process gives you a complete map of all the pieces.

Steps to add sys_hello on x86-64:

1. Define the implementation in an appropriate subsystem file (e.g., kernel/sys.c):

SYSCALL_DEFINE1(hello, int, val)
{
    printk(KERN_INFO "hello syscall: val=%d\n", val);
    return val * 2;
}

2. Assign a number in arch/x86/entry/syscalls/syscall_64.tbl:

451  common  hello  sys_hello

3. Declare the prototype in include/linux/syscalls.h:

asmlinkage long sys_hello(int val);

4. Add the generic entry in include/uapi/asm-generic/unistd.h if you want the syscall on all architectures.

5. Wire up compat (for 32-bit processes on 64-bit kernels) if your syscall has pointer or size_t arguments that differ between 32/64-bit.

6. Test it with a simple userspace program calling syscall(451, 42) on a custom-built kernel.

7. Write a test for tools/testing/selftests/ — the kernel test suite.

What the kernel community will ask:

• Is this really needed, or can it be done with existing syscalls?
• Is the interface future-proof? (Wrong interfaces become permanent debt.)
• Does it have proper __user annotations and copy_from_user for all pointers?
• Does it work on all architectures?
• Is it documented in man2/ (the man-pages project)?

The clone3 syscall (2019) is a great case study: it replaced the messy flag-overloaded clone with a struct-based interface specifically to be extensible.

// Test your new sys_hello from userspace:
#include <sys/syscall.h>
#include <stdio.h>

#define SYS_hello 451  // the number you assigned

int main(void) {
    long ret = syscall(SYS_hello, 21);
    printf("sys_hello(21) = %ld\n", ret);  // should print 42
    return 0;
}