System Call Implementation in the Kernel

Modern CPUs enforce isolation between programs using hardware privilege rings (x86) or exception levels (ARM). User-space code runs at ring 3 (x86) or EL0 (ARM), where instructions that touch hardware directly — like reading from an I/O port or modifying page tables — trigger a fault. The kernel runs at ring 0 / EL1, where everything is permitted.

The only architecturally sanctioned way to cross this boundary upward (from unprivileged to privileged) is through a synchronous exception: on x86-64 the syscall instruction, on ARM64 the svc #0 instruction, on RISC-V the ecall instruction. Each of these causes the CPU to atomically:

1. Save the current program counter and CPU flags to well-known registers or a kernel stack.
2. Switch the privilege level.
3. Jump to a fixed kernel-supplied handler address that was previously written into a CPU register (x86-64: the LSTAR MSR; ARM64: the VBAR_EL1 vector base register).

Notice what the CPU does NOT do: it does not trust any pointer or address that came from user space. It jumps to a kernel-controlled address only. This is the hardware root of all kernel security. The kernel entry code then reads the syscall number from a register (rax on x86-64, x8 on ARM64, a7 on RISC-V) and uses it to dispatch to the right handler. That dispatch table — and the macros that populate it — is what the rest of this module is about.

/* How glibc issues a 'write' syscall on x86-64:
   rax = syscall number (1 = __NR_write)
   rdi = fd, rsi = buf, rdx = count          */
mov    $1,   %rax      /* __NR_write           */
mov    $1,   %rdi      /* stdout               */
lea    msg,  %rsi      /* pointer to buffer    */
mov    $13,  %rdx      /* byte count           */
syscall               /* trap into kernel     */
/* return value (or -errno) lands in rax      */

When the syscall instruction fires, the CPU is now in ring 0 but the stack pointer still points at the user stack — a pointer that must never be trusted. The very first thing the entry code does is switch to the kernel stack for this thread. On x86-64 this is done by the swapgs instruction (which swaps the GS base register so the per-CPU area becomes visible) followed by loading the kernel stack pointer from cpu_entry_area.

The entry trampoline (arch/x86/entry/entry_64.S, function entry_SYSCALL_64) then:

1. Saves all callee-saved registers (the full pt_regs structure) onto the kernel stack.
2. Calls the C function do_syscall_64(struct pt_regs *regs, int nr).
3. On return, checks whether the process has pending signals or needs rescheduling (the syscall slow path).
4. Restores registers and executes sysretq to return to user space.

The pt_regs structure is crucial: it is the snapshot of the user-space CPU state at the moment of the trap. If you have ever seen a kernel stack trace that shows entry_SYSCALL_64 at the bottom, now you know why — it is the outermost frame, the trampoline.

do_syscall_64 does one thing elegantly: it looks up sys_call_table[nr] (after bounds-checking and security audits like seccomp) and calls the function pointer. The return value from that function is written into regs->ax, which becomes the return value seen by user space.

/* Simplified do_syscall_64 — real code: arch/x86/entry/common.c */
__visible noinstr void do_syscall_64(struct pt_regs *regs, int nr)
{
    add_random_kstack_offset();       /* stack canary entropy   */
    nr = syscall_enter_from_user_mode(regs, nr); /* seccomp etc */

    if (likely(nr < NR_syscalls))
        regs->ax = sys_call_table[nr](regs); /* dispatch! */
    else
        regs->ax = -ENOSYS;

    syscall_exit_to_user_mode(regs);  /* signals, rescheduling  */
}

The syscall table is the simplest possible data structure: a plain array of function pointers, indexed by syscall number. On x86-64 it lives in arch/x86/entry/syscall_64.c and is generated by the build system from a table definition file (arch/x86/entry/syscalls/syscall_64.tbl).

Each line in syscall_64.tbl looks like:

1   common  write       sys_write

The number (1) is the ABI index. The string common means the handler is shared between 64-bit native and x32 ABI. sys_write is the C function name. The build system runs syscalltbl.sh which generates a C file that produces sys_call_table[].

On ARM64 and RISC-V the mechanism is similar but the table is architecture-independent and lives in include/uapi/asm-generic/unistd.h, populated by a different macro scheme.

Why a plain array? Because syscall dispatch happens on every single I/O operation — it must be as fast as possible. An array indexed by an integer is O(1) with a single bounds check. The table is marked const, lives in read-only memory after boot (with __ro_after_init), and is protected by kernel page protections against runtime modification — a critical security hardening against attackers who might try to hijack the table.

/* arch/x86/entry/syscall_64.c (generated, simplified) */
asmlinkage const sys_call_ptr_t sys_call_table[] = {
    [0]  = sys_read,
    [1]  = sys_write,
    [2]  = sys_open,
    /* ... 400+ entries ... */
    [NR_syscalls-1] = sys_ni_syscall, /* not-implemented stub */
};

/* Entries with no handler use this safe stub */
asmlinkage long sys_ni_syscall(void) { return -ENOSYS; }

Every system call implementation you read in the kernel uses a macro like SYSCALL_DEFINE3(write, ...) rather than writing asmlinkage long sys_write(...) by hand. This seems like mere style, but there are several important reasons:

1. Argument tracing. The SYSCALL_DEFINEn macros (where n is the argument count, 0–6) automatically instrument the function for ftrace / perf syscall tracepoints. When you run perf trace ls, the kernel fires a tracepoint at every syscall entry and exit — the macro wires that up for free.

2. Type safety on 32-bit compat paths. On a 64-bit kernel running 32-bit user-space binaries, arguments arrive in 32-bit registers. The macros generate both a native 64-bit handler and (if needed) a compat handler, ensuring argument widths are correct.

3. Seccomp argument inspection. seccomp(2) with BPF filters needs to inspect individual syscall arguments. The macro expansion stores argument metadata in a form seccomp can use.

4. Stack protector hints. The macros add __MAP and __SC_DECL helpers that the compiler can use to reason about argument provenance.

The macro expands into two things: the actual function body (named __se_sys_NAME), and a thin wrapper (__do_sys_NAME) that unpacks the pt_regs struct into named parameters. On x86-64, modern kernels pass struct pt_regs * to the table entry, and the wrapper extracts rdi, rsi, rdx, etc. from it. This is called the regparm-based calling convention for syscalls introduced around kernel 4.17.

/* How SYSCALL_DEFINE3 expands — conceptually */

/* What you WRITE: */
SYSCALL_DEFINE3(write, unsigned int, fd,
                const char __user *, buf, size_t, count)
{
    return ksys_write(fd, buf, count);
}

/* What the MACRO expands to (simplified): */
asmlinkage long sys_write(const struct pt_regs *regs);
static long __se_sys_write(unsigned int fd,
                           const char __user *buf, size_t count);
static inline long __do_sys_write(unsigned int fd,
                                  const char __user *buf, size_t count)
{
    return ksys_write(fd, buf, count);
}
asmlinkage long sys_write(const struct pt_regs *regs) {
    return __se_sys_write((unsigned int)regs->di,
                          (const char __user *)regs->si,
                          (size_t)regs->dx);
}
static long __se_sys_write(unsigned int fd,
                           const char __user *buf, size_t count) {
    long ret = __do_sys_write(fd, buf, count);
    /* tracepoint, audit hooks fire here */
    return ret;
}

This is perhaps the single most important concept in kernel programming for security. When a syscall receives a pointer from user space, that pointer could be:

• Pointing to kernel memory (a classic privilege escalation vector).
• Pointing to unmapped memory (would cause a kernel oops).
• Changing underneath you (TOCTOU: time-of-check to time-of-use race condition).
• Pointing to a memory-mapped device register.

The kernel therefore has a strict rule: you must never dereference a user-space pointer directly. You must use the copy_from_user(kernel_buf, user_ptr, size) and copy_to_user(user_ptr, kernel_buf, size) functions (or their scalar variants get_user / put_user).

These functions do several things:

1. Access_ok check: Verify that the entire range [user_ptr, user_ptr+size) lies within the valid user address space for this process (below TASK_SIZE). This rejects kernel-address pointers.
2. Fault-tolerant copy: The actual memory copy uses a special code path that installs a fixup in the kernel's exception table. If a page fault occurs during the copy (because the user page is not resident or the mapping is invalid), the CPU's page-fault handler checks the exception table, finds the fixup for that instruction, and returns a partial-copy error rather than crashing the kernel. The function returns the number of bytes NOT copied (0 on full success).
3. SMAP/PAN enforcement: On x86-64 with SMAP (Supervisor Mode Access Prevention) and ARM64 with PAN (Privileged Access Never), the CPU hardware will fault if kernel code touches user pages without going through these functions, which temporarily clear the protection bit.

Always check the return value of copy_from_user. A non-zero return means the copy failed partway; you must return -EFAULT to user space.

#include <linux/uaccess.h>

/* Imagine a syscall that reads a 'struct point' from user space */
struct point { long x; long y; };

SYSCALL_DEFINE1(my_point_abs, struct point __user *, uptr)
{
    struct point kp;   /* kernel-side copy — lives on kernel stack */
    long dist;

    /* NEVER: kp = *uptr;  -- direct deref of user pointer! */

    if (copy_from_user(&kp, uptr, sizeof(kp)))
        return -EFAULT;   /* partial or failed copy */

    /* Now kp is safe to use */
    dist = abs(kp.x) + abs(kp.y);
    return dist;  /* returned in rax to user space */
}

/* For single primitive values, prefer get_user / put_user */
int val;
if (get_user(val, (int __user *)uptr))
    return -EFAULT;

Beyond pointer safety, syscall implementations must validate every argument for semantic correctness before acting on it. This is a discipline that appears throughout the kernel and is worth studying explicitly.

File descriptors: fdget() / fdget_pos() look up a file descriptor in the current process's file descriptor table and return a reference-counted struct fd. If the fd is out of range or closed, they return an empty struct. Always check fd_empty(f) before using.

Sizes and counts: Never trust a user-supplied size without bounds checking. A size argument of SIZE_MAX passed to an allocation is an integer overflow waiting to happen. Check size > MAX_RW_COUNT for read/write, check for count == 0 (return 0 immediately), etc.

Flags and mode bits: Mask off unknown flags with ~(KNOWN_FLAGS) and return -EINVAL if any unknown bits are set. This preserves forward compatibility: a future kernel can define a new flag, and old kernels reject it rather than silently ignoring it.

Alignment: Some hardware or kernel structures require naturally aligned addresses. Check with IS_ALIGNED(addr, sizeof(type)).

The rule of thumb: validate eagerly, act late. Do all checks at the top of the function before modifying any kernel state. This makes error paths clean and prevents partial-mutation bugs where the kernel state is corrupted if a later check fails.

The __user annotation (from include/linux/compiler_types.h) is not enforced at runtime — it is a semantic tag for the Sparse static-analysis tool (make C=1). Sparse will warn if you dereference a __user pointer without copy_from_user. Run it on your code.

SYSCALL_DEFINE3(my_bounded_read,
    int, fd, char __user *, buf, size_t, count)
{
    struct fd f;
    ssize_t ret;

    /* 1. Trivial fast path */
    if (count == 0)
        return 0;

    /* 2. Reject absurdly large requests */
    if (count > MAX_RW_COUNT)
        count = MAX_RW_COUNT;   /* or return -EINVAL */

    /* 3. Validate fd */
    f = fdget_pos(fd);
    if (fd_empty(f))
        return -EBADF;

    /* 4. Check file is readable */
    if (!(f.file->f_mode & FMODE_READ)) {
        ret = -EBADF;
        goto out_fput;
    }

    /* 5. buf pointer validated inside vfs_read -> copy_to_user */
    ret = vfs_read(f.file, buf, count, &f.file->f_pos);

out_fput:
    fdput_pos(f);
    return ret;
}

When a thread enters the kernel via a syscall, it switches from its user-space stack to a dedicated kernel stack. On x86-64, each thread has an 8 KB (two pages) kernel stack allocated when the thread is created. On ARM64 it is 16 KB.

This is a very small stack. This is intentional and important:

• It bounds worst-case memory usage (N threads × 8 KB, not N × 8 MB).
• It prevents user space from arbitrarily consuming kernel memory by simply creating threads.
• It forces kernel code to be stack-disciplined: no large on-stack buffers, no deep recursion.

The pt_regs structure (containing the saved user-space register state) is placed at the top of the kernel stack by the entry trampoline. The rest of the 8 KB is available for the call tree from do_syscall_64 downward. If that tree overflows — CONFIG_VMAP_STACK places a guard page below the stack — the CPU will double-fault. CONFIG_KASAN and CONFIG_STACK_VALIDATION can detect stack overflows at build or runtime.

The current macro (actually an inline function using the gs segment register on x86-64) gives kernel code a pointer to the task_struct of the currently executing thread at any point during syscall execution. This is how, for example, copy_from_user knows which process's memory map to check — it calls access_ok() which consults current->mm.

/*
 * Kernel stack layout during a syscall on x86-64
 * (addresses grow downward)
 *
 *  [high addr]  top of kernel stack page
 *               pt_regs  (user rip, rsp, rax, rbx ... saved here)
 *               do_syscall_64 frame
 *               sys_write frame
 *               ksys_write frame
 *               vfs_write frame
 *               ... deeper call tree ...
 *  [low addr]   guard page (unmapped, triggers double fault if hit)
 *
 * Total budget: ~8 KB. Never put large arrays here.
 */

/* 'current' gives you the task_struct of the running thread */
#include <linux/sched.h>
printk(KERN_DEBUG "syscall by pid %d (%s)\n",
       current->pid, current->comm);

The fast path through a syscall is: enter → dispatch → run handler → return. But there is a slow path that activates when the kernel needs to do extra work on the way back to user space.

syscall_exit_to_user_mode() checks a bitmask of pending work flags on the current thread before returning:

• TIF_SIGPENDING: A signal is queued. The signal handler must be invoked before returning to the original user-space instruction.
• TIF_NEED_RESCHED: The scheduler wants to run a different thread. schedule() is called here.
• TIF_NOTIFY_RESUME: ptrace, seccomp notification, or user-mode instruction extensions need attention.

This is also where restartable syscalls come into play. If a blocking syscall like read() on a pipe is interrupted by a signal, the handler must return either -EINTR (the call was interrupted, user sees the error) or -ERESTARTSYS (the kernel will automatically re-issue the syscall after the signal handler returns, transparently to the user). The ERESTARTSYS / ERESTARTNOINTR / ERESTARTNOHAND codes are kernel-internal; the signal delivery path in arch/x86/kernel/signal.c decides whether to rewind the user's instruction pointer to re-execute the syscall instruction.

This machinery is why read() on a pipe can sometimes return fewer bytes than requested without it being an error — and why wrapper libraries like glibc loop on EINTR.

/* A simplified interruptible wait — from kernel driver code */
long my_blocking_syscall_body(void)
{
    DEFINE_WAIT(wait);
    long ret = 0;

    add_wait_queue(&my_wq, &wait);
    for (;;) {
        prepare_to_wait(&my_wq, &wait, TASK_INTERRUPTIBLE);

        if (data_is_ready())
            break;

        if (signal_pending(current)) {
            ret = -ERESTARTSYS;  /* kernel will restart us */
            break;
        }
        schedule();  /* yield CPU, wake up when signaled or ready */
    }
    finish_wait(&my_wq, &wait);
    return ret;
}

Adding a syscall to the Linux kernel is intentionally bureaucratic — the committee reviews new syscalls with extreme scrutiny because of the ABI stability promise. Nonetheless, doing it yourself in a local kernel build is the single best learning exercise in this module. Here is the complete recipe for x86-64 on a v6.x kernel:

Step 1 — Add the number to arch/x86/entry/syscalls/syscall_64.tbl:

463   common   my_hello    sys_my_hello

(Pick a number not already used; check by grepping the tbl file.)

Step 2 — Declare it in include/linux/syscalls.h:

asmlinkage long sys_my_hello(int magic);

Step 3 — Implement it in a new file, e.g., kernel/my_hello.c:

#include <linux/syscalls.h>
#include <linux/printk.h>
SYSCALL_DEFINE1(my_hello, int, magic) {
    if (magic != 42)
        return -EINVAL;
    pr_info("hello from syscall, pid=%d\n", current->pid);
    return 0;
}

Step 4 — Wire it into the build by adding to kernel/Makefile:

obj-y += my_hello.o

Step 5 — Test it from user space:

#include <sys/syscall.h>
#include <unistd.h>
#include <stdio.h>
#define __NR_my_hello 463
int main(void) {
    long r = syscall(__NR_my_hello, 42);
    printf("ret=%ld\n", r);  /* should print 0 */
    return 0;
}

Step 6 — Add a kselftest under tools/testing/selftests/syscall_user_dispatch/ or a new directory. Real upstream submissions require this.

For real upstream submission: you also need a man2 page draft, an entry in include/uapi/asm-generic/unistd.h (for non-x86 arches), compat handlers if any argument is 32-bit-sensitive, and a thorough cover letter justifying why existing syscalls cannot do the job.

/* kernel/my_hello.c */
#include <linux/syscalls.h>
#include <linux/printk.h>
#include <linux/sched.h>

SYSCALL_DEFINE1(my_hello, int, magic)
{
    if (magic != 42)
        return -EINVAL;

    pr_info("my_hello: called by pid %d (%s) with magic=%d\n",
            current->pid, current->comm, magic);
    return 0;
}

/* tools/testing/selftests/my_hello/my_hello_test.c */
#include <sys/syscall.h>
#include <unistd.h>
#include <assert.h>
#define __NR_my_hello 463
int main(void) {
    assert(syscall(__NR_my_hello, 42)  == 0);
    assert(syscall(__NR_my_hello, 99)  == -1); /* errno=EINVAL */
    return 0;
}

The Linux kernel has one of the most serious compatibility promises in all of software: we do not break userspace. Linus Torvalds has rejected patches that break the syscall ABI in terms that leave no ambiguity. The reasoning is profound: every binary ever compiled and distributed — in a Docker image, a proprietary application, a 10-year-old embedded system — may rely on a syscall's exact number, argument layout, and semantics. The kernel is the only piece of software that must be forward-compatible with all of them.

This promise means:

• Syscall numbers never change or get reused. A removed syscall's slot is replaced with sys_ni_syscall (the not-implemented stub) forever. The slot is never reassigned to a different syscall.
• Argument types and orders are frozen. You can add behavior via flags, but you cannot reorder arguments or change their meaning.
• New flags must be zero-or-fail. Any new flag bit you add must be rejected by old kernels (which return -EINVAL for unknown flags) and accepted by new ones, never silently ignored.
• Struct layouts passed across the boundary are frozen (think stat, sigaction, timespec). This is why the kernel has statx(2) (which uses a size field to version the struct) rather than extending stat(2).
• The UAPI headers (include/uapi/) are specifically separated from internal kernel headers precisely because they form the public ABI. Changes there are reviewed with extra care.

For a learner, this means: when you read kernel code that looks needlessly complex or has multiple versions of a struct, the reason is almost always ABI stability. The complexity is the price of 30+ years of compatibility.

/* WHY statx exists instead of extending stat:

   struct stat has a fixed layout from 1970s Unix.
   You cannot add fields — old binaries compiled against
   the old layout would read garbage from new fields.

   statx() solves this with an explicit 'struct_size' field:
*/

/* User tells kernel how big their struct is */
int ret = statx(AT_FDCWD, path,
                AT_STATX_SYNC_AS_STAT,
                STATX_ALL,
                &statxbuf);  /* struct statx, 256 bytes */

/* The kernel only fills fields within the requested size.
   A future kernel can grow struct statx; old binaries are safe
   because they only passed in sizeof(old_struct_statx). */

/* Slots for retired syscalls are kept as ni_syscall forever */
/* From arch/x86/entry/syscalls/syscall_64.tbl: */
/* 59 is execve -- never reassigned, never removed */
/* 134 was sys_uselib -- now ni_syscall, slot preserved */

Two tools are indispensable when writing syscall code:

Sparse (make C=1) is a semantic checker that understands kernel type annotations like __user, __kernel, __rcu, __percpu. It will warn if you dereference a __user pointer directly (a security bug), mix address spaces, or annotate function pointers incorrectly. It is the first line of defense against copy_from_user omissions. Run it on every file you touch.

ftrace / perf trace let you observe syscall entry and exit in a running kernel. The SYSCALL_DEFINE macros automatically register tracepoints at syscalls:sys_enter_NAME and syscalls:sys_exit_NAME. You can observe them with:

sudo perf trace -e 'syscalls:sys_enter_write' ls

or with the ftrace interface:

echo 'syscalls:sys_enter_write' > /sys/kernel/tracing/set_event
cat /sys/kernel/tracing/trace_pipe

For your custom syscall, the tracepoint syscalls:sys_enter_my_hello is automatically created by the SYSCALL_DEFINE macro — you get observability for free.

strace works at the ptrace level (the kernel notifies a tracer process at each syscall entry/exit via PTRACE_SYSCALL). It is slower than ftrace but easier to use from userspace without root. Understanding that strace is implemented with ptrace — not with ftrace tracepoints — teaches you that there are multiple independent observation planes for the same event.

# Build with Sparse checking (in your kernel source tree)
make C=1 kernel/my_hello.o
# Look for: "warning: incorrect type in argument" (missing __user)
# or: "warning: dereference of noderef expression" (raw user ptr)

# Observe your syscall live with ftrace
cd /sys/kernel/tracing
echo 1 > tracing_on
echo 'syscalls:sys_enter_my_hello' > set_event
# In another terminal: run your test binary
cat trace_pipe
# Output: <...>-1234 [000] .... my_hello: magic=42

# perf stat for syscall frequency
perf stat -e 'syscalls:sys_enter_read,syscalls:sys_enter_write' \
    dd if=/dev/zero of=/dev/null bs=4k count=10000