eBPF: The Programmable Kernel

At its core, eBPF defines a simple RISC-style virtual machine. It has 11 registers (R0–R10), each 64 bits wide. R1–R5 carry function arguments. R0 holds the return value. R10 is the read-only frame pointer pointing to the BPF stack (at most 512 bytes). R6–R9 are callee-saved temporaries. There are ~100 instructions grouped into arithmetic/logic, move, load/store, jump, and call families. The ISA was deliberately kept minimal so the verifier and JIT could be written simply and correctly.

When you compile a BPF program with Clang (clang -target bpf -O2 -c), you get a .o ELF file containing BPF bytecode — the same instruction set regardless of the host CPU. When the program is loaded into the kernel via bpf(BPF_PROG_LOAD, ...), two things happen in sequence:

1. The verifier runs (see next concept). If it passes, the bytecode is handed to the JIT.
2. The JIT compiler translates BPF bytecode to native machine code for the host architecture — x86-64, arm64, riscv64, s390, etc. After JIT, the BPF program runs at native speed, not interpreted speed. There is essentially zero overhead vs. a hand-written kernel function.

You can inspect the JIT output: echo 1 > /proc/sys/net/core/bpf_jit_enable then load a program and look at /sys/kernel/debug/tracing/trace_pipe or use bpftool prog dump jited id <N>. This is a great exercise to see real x86-64 code that was generated from your C.

The BPF stack is only 512 bytes — tiny by user-space standards, but it forces BPF programs to be lean. Large data goes into BPF maps (shared key-value stores in kernel memory). Calls between BPF functions (BPF-to-BPF calls) and tail calls (jump to another BPF program, resetting the stack) allow programs to be composed without blowing the 512-byte limit.

/* minimal BPF program: count XDP packets (raw bytecode view, but
   the C source that clang compiles to this bytecode follows) */

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

/* A per-CPU array map with 1 element: our packet counter */
struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} pkt_count SEC(".maps");

SEC("xdp")
int count_packets(struct xdp_md *ctx)
{
    __u32 key = 0;
    __u64 *val = bpf_map_lookup_elem(&pkt_count, &key);
    if (val)
        __sync_fetch_and_add(val, 1);
    return XDP_PASS;  /* let the packet through */
}

char _license[] SEC("license") = "GPL";

The BPF verifier is the most intellectually interesting part of the whole system. It lives in kernel/bpf/verifier.c and is roughly 30,000 lines of code — one of the largest single files in the kernel. Before any BPF program executes even once, the verifier performs an exhaustive simulation of all possible execution paths through the program.

What the verifier checks:

• Termination: the program's control-flow graph must be a DAG (no backward edges unless the kernel version supports bounded loops, which were added in 5.3 with explicit loop count verification).
• Memory safety: every load and store must be to a provably valid address — a BPF stack slot, a BPF map value, a context field within the declared bounds, or a kernel pointer returned by a helper and known to be non-null.
• Type safety: the verifier tracks the type of every register at every program point. R1 might be a PTR_TO_CTX, R2 a PTR_TO_MAP_VALUE_OR_NULL, R0 a SCALAR_VALUE. Dereferencing a SCALAR_VALUE as a pointer is rejected.
• Null checks: if a helper returns a pointer-or-null (e.g. bpf_map_lookup_elem), the verifier requires a null check before the pointer is dereferenced. This is the source of the most common beginner error: forgetting if (!val) return; before using a map lookup result.
• Helper call ABI: each helper function has a declared prototype. The verifier checks argument types and marks the return value with the correct type.

The verifier works by tracking abstract state — for each register, it knows whether it's uninitialized, a known constant, a range of values, or a typed pointer — and propagating that state along every edge of the CFG. This is essentially abstract interpretation / dataflow analysis, the same technique used in formal verification tools. When a branch is taken, the verifier forks the state and explores both paths.

A key concept is the reference counter: some helpers (like bpf_sk_lookup_tcp) return a reference that must be released before the program exits. The verifier tracks outstanding references and rejects programs that leak them.

When the verifier rejects a program, strace bpf(...) shows the errno and the kernel log contains a human-readable explanation of exactly which instruction failed and why. Reading these messages is a core debugging skill.

/* This program will be REJECTED by the verifier.
   Can you spot why before reading the comment? */

SEC("xdp")
int bad_program(struct xdp_md *ctx)
{
    __u32 key = 0;
    __u64 *val = bpf_map_lookup_elem(&pkt_count, &key);
    /* BUG: val might be NULL — verifier will reject the next line
       with "R0 invalid mem access 'map_value_or_null'" */
    *val += 1;
    return XDP_PASS;
}

/* Fixed version: */
SEC("xdp")
int good_program(struct xdp_md *ctx)
{
    __u32 key = 0;
    __u64 *val = bpf_map_lookup_elem(&pkt_count, &key);
    if (!val) return XDP_PASS;  /* null check satisfies verifier */
    *val += 1;
    return XDP_PASS;
}

BPF programs run in the kernel but need to communicate with userspace (to export metrics, receive configuration, share state across invocations). That channel is BPF maps: typed key-value stores that live in kernel memory and can be accessed from both BPF programs (via helpers) and userspace processes (via bpf(BPF_MAP_LOOKUP_ELEM, ...) syscall).

Maps are created with bpf(BPF_MAP_CREATE, ...) specifying the type, key size, value size, and max entries. The kernel returns a file descriptor; the fd is the reference-counted handle to the map. Maps persist as long as some fd (or pinned path in /sys/fs/bpf/) holds a reference — they outlive the program that created them.

Important map types:

• BPF_MAP_TYPE_HASH: general-purpose hash table. O(1) average lookup. Used for flow tables, per-connection state.
• BPF_MAP_TYPE_ARRAY: fixed-size array indexed by u32. All elements pre-allocated. Used for counters, configuration.
• BPF_MAP_TYPE_PERCPU_ARRAY / BPF_MAP_TYPE_PERCPU_HASH: each CPU has its own copy of the value. Eliminates contention for high-frequency per-CPU counters. Userspace reads all CPU copies and sums them.
• BPF_MAP_TYPE_RINGBUF (5.8+): a lock-free single-producer single-consumer ring buffer. The preferred way to send events from BPF to userspace with low overhead. Replaces the older perf_event output mechanism.
• BPF_MAP_TYPE_LRU_HASH: hash with automatic eviction of least-recently-used entries. Essential for flow tables that must not grow unboundedly.
• BPF_MAP_TYPE_PROG_ARRAY: an array of BPF program FDs. Used for tail calls — bpf_tail_call(ctx, &prog_array, index) jumps to another program, resetting the stack. The maximum tail-call chain depth is 33.
• BPF_MAP_TYPE_SOCKHASH / BPF_MAP_TYPE_SOCKMAP: maps of sockets, used for socket redirection (sk_msg / sk_skb programs can redirect packets between sockets without going through the network stack).
• BPF_MAP_TYPE_STRUCT_OPS: allows BPF programs to implement kernel ops structs — for example, implementing a custom TCP congestion control algorithm entirely in BPF.

Map pinning is how you share a map between multiple processes or make it persist across program restarts: bpf(BPF_OBJ_PIN, ...) creates a path in the BPF virtual filesystem (/sys/fs/bpf/) that holds a reference. Another process can open that path with bpf(BPF_OBJ_GET, ...) and get a new fd to the same map.

/* Userspace: create a hash map and read entries after BPF runs */
#include <bpf/libbpf.h>
#include <bpf/bpf.h>
#include <stdio.h>

int main(void) {
    /* In practice, libbpf skeleton does this automatically.
       Manual API shown for clarity. */
    int map_fd = bpf_map_create(BPF_MAP_TYPE_HASH,
                                "demo_map",
                                sizeof(__u32),   /* key size */
                                sizeof(__u64),   /* value size */
                                1024,            /* max entries */
                                NULL);
    if (map_fd < 0) { perror("map_create"); return 1; }

    /* Pin it so the BPF program (loaded separately) can find it */
    bpf_obj_pin(map_fd, "/sys/fs/bpf/demo_map");

    /* ... load and attach BPF program ... */

    /* Read a value back */
    __u32 key = 42;
    __u64 value;
    if (bpf_map_lookup_elem(map_fd, &key, &value) == 0)
        printf("key=%u value=%llu\n", key, value);
    return 0;
}

A BPF program is useless until it is attached to a hook — a well-defined point in the kernel where the program is called with a typed context. The hook determines the program type, the context struct the program receives, what helpers are available, and what the return value means. Understanding the hook landscape is understanding the power of eBPF.

XDP (eXpress Data Path): Runs at the earliest possible point in the network receive path, inside the device driver, before the kernel allocates a socket buffer (sk_buff). The context is struct xdp_md with pointers to the raw packet bytes. Return values: XDP_PASS (continue normally), XDP_DROP (drop immediately — no copy, no log), XDP_TX (send back out the same interface), XDP_REDIRECT (send to another interface or CPU). XDP is used for DDoS mitigation (drop malicious packets at line rate, ~10 ns/packet vs. ~1 µs in iptables), load balancing (Katran at Meta, Cilium), and high-performance packet forwarding. It requires driver support (native XDP) or falls back to generic XDP (slower, runs after sk_buff allocation).

TC (Traffic Control) hooks: tc programs attach to the cls_bpf classifier in the Linux traffic control (qdisc) layer, running on both ingress and egress. Unlike XDP (receive only), TC can intercept outgoing packets. The context is the full sk_buff. Used in Cilium for container-level packet filtering and NAT.

Socket hooks: BPF_PROG_TYPE_SOCK_OPS can intercept TCP state machine transitions (connection established, etc.) to set socket options, implement SYN cookies, or route connections. BPF_PROG_TYPE_SK_SKB and BPF_PROG_TYPE_SK_MSG allow in-kernel socket-level redirection — useful for service meshes to avoid kernel/userspace copies.

Tracepoints: Static hooks compiled into the kernel at specific interesting points (scheduler events, syscalls, block I/O, etc.). The kernel defines a stable ABI for each tracepoint. BPF programs attached here receive a context struct defined by the tracepoint format. bpftrace and perf use these heavily.

kprobes and kretprobes: Dynamic hooks inserted at any kernel function entry or return, without recompilation. The kernel patches a int3 (on x86) at the target instruction; when it fires, the BPF program runs. Incredibly powerful for debugging and profiling, but not ABI-stable — function signatures can change between kernel versions. This is one reason CO-RE (see later concept) matters.

uprobes: Same mechanism as kprobes but for userspace functions. Attach to a function in libc, a JVM, a Python interpreter — anything. Used by bpftrace to trace user programs without recompilation.

LSM hooks (BPF_PROG_TYPE_LSM, added in 5.7): BPF programs that implement Linux Security Module hooks. Allows writing kernel security policies in BPF — KRSI (Kernel Runtime Security Instrumentation). Programs can allow or deny operations like file_open, socket_connect, task_kill. This is how Google's Landlock (and production security tools like Falco) work.

/* XDP program: drop all UDP packets on port 53 (toy DNS block) */
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

SEC("xdp")
int block_dns(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end) return XDP_PASS;
    if (eth->h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end) return XDP_PASS;
    if (ip->protocol != IPPROTO_UDP)  return XDP_PASS;

    struct udphdr *udp = (void *)(ip + 1);
    if ((void *)(udp + 1) > data_end) return XDP_PASS;

    if (udp->dest == bpf_htons(53)) return XDP_DROP;
    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

bpftrace is a high-level tracing language that compiles scripts to BPF programs on the fly. Think of it as awk for kernel events — a one-liner can give you production-grade tracing in seconds. It was heavily inspired by DTrace (the tracing framework from Solaris) and built on top of libbpf and LLVM.

A bpftrace program is a series of probes, each with the form probe_spec { action }. Probe specs include:

• kprobe:do_sys_open — attach to kernel function entry
• kretprobe:do_sys_open — attach to kernel function return
• tracepoint:syscalls:sys_enter_read — attach to a tracepoint
• uprobe:/lib/libc.so.6:fopen — attach to a userspace function
• profile:99hz — sample all CPUs at 99 Hz (for flame graphs)
• interval:s:1 — fire once per second (for periodic output)

Built-in variables: pid, tid, comm (process name), nsecs (nanoseconds since boot), args (tracepoint argument struct), retval (kretprobe return value), curtask (pointer to struct task_struct).

Built-in maps: @mymap[key] = value — bpftrace automatically creates a BPF hash map. @latency = hist(nsecs) creates a log2 histogram. count() is a shorthand for @count++.

The beauty of bpftrace for learning is the immediacy: you write a three-line script, it compiles to BPF, loads into the kernel, runs, and you see results in real time. You can explore kernel internals interactively without writing a single line of libbpf C code.

#!/usr/bin/env bpftrace

// Measure read() syscall latency distribution, per process
// Run: sudo bpftrace latency.bt

tracepoint:syscalls:sys_enter_read
{
    @start[tid] = nsecs;
}

tracepoint:syscalls:sys_exit_read
/@start[tid]/
{
    @latency_ns[comm] = hist(nsecs - @start[tid]);
    delete(@start[tid]);
}

END
{
    // printed automatically at Ctrl-C
}

For production BPF programs — where you need full control, structured build systems, and reliable lifecycle management — the tool is libbpf: the official C library for loading, attaching, and interacting with BPF programs. It lives in tools/lib/bpf/ in the kernel tree and is also mirrored at https://github.com/libbpf/libbpf.

The modern workflow uses BPF skeletons, auto-generated by bpftool gen skeleton. You write the BPF C file, compile it to a .bpf.o, run bpftool gen skeleton prog.bpf.o > prog.skel.h, and get a header with:

• prog_bpf__open() — opens the ELF, reads BTF, sets up maps
• prog_bpf__load() — verifies and JIT-compiles all programs
• prog_bpf__attach() — attaches programs to their declared hooks
• prog_bpf__destroy() — cleans up
• Direct struct access: skel->maps.pkt_count, skel->progs.count_packets, skel->data->global_var

This makes the userspace loader almost trivially simple — the skeleton handles the plumbing. The BPF program's maps and programs are accessible as typed C struct fields, so you get compile-time type checking for map access from userspace.

For sending events to userspace efficiently, the ring buffer pattern is standard: the BPF program calls bpf_ringbuf_output() or bpf_ringbuf_reserve()/bpf_ringbuf_submit(), and userspace calls ring_buffer__poll() in a loop, which invokes a callback for each event. Latency is in the microseconds range; throughput is millions of events per second.

/* userspace loader using skeleton (auto-generated header) */
#include "count_packets.skel.h"
#include <stdio.h>
#include <unistd.h>
#include <net/if.h>

int main(int argc, char **argv) {
    struct count_packets_bpf *skel;
    int err, ifindex;

    ifindex = if_nametoindex(argc > 1 ? argv[1] : "eth0");

    skel = count_packets_bpf__open_and_load();
    if (!skel) { fprintf(stderr, "load failed\n"); return 1; }

    /* attach XDP to the interface */
    skel->links.count_packets =
        bpf_program__attach_xdp(skel->progs.count_packets, ifindex);

    printf("Counting packets on ifindex %d. Ctrl-C to stop.\n", ifindex);
    for (;;) {
        sleep(1);
        __u32 key = 0;
        __u64 total = 0, val;
        int ncpus = libbpf_num_possible_cpus();
        __u64 values[ncpus];
        bpf_map__lookup_elem(skel->maps.pkt_count,
                             &key, sizeof(key),
                             values, sizeof(values), 0);
        for (int i = 0; i < ncpus; i++) total += values[i];
        printf("packets/sec: %llu\n", total);
    }

    count_packets_bpf__destroy(skel);
    return 0;
}

Here is a fundamental problem: a BPF program compiled on kernel 5.15 that reads task->pid at byte offset 1234 from a struct task_struct will corrupt data or crash on kernel 5.19 if the struct layout changed (which it regularly does). For years, BPF programs had to be compiled on the target machine to match its exact kernel headers. This made distribution impossible.

BTF (BPF Type Format) is a compact binary encoding of C type information — structs, unions, enums, typedefs — embedded in the kernel image itself (see /sys/kernel/btf/vmlinux) and in BPF object files. Think of BTF as DWARF (debug info), but designed to be small enough to include in a production kernel and efficient enough for the verifier to query at runtime.

CO-RE (Compile Once — Run Everywhere) uses BTF to make BPF programs portable. The key mechanism is BPF field relocations: when your BPF program accesses task->pid, Clang emits a relocation record that says "I want the field named pid in struct task_struct". At load time, libbpf reads the target kernel's BTF from /sys/kernel/btf/vmlinux, looks up where pid actually lives in that kernel's layout, and patches the BPF bytecode with the correct offset before verification. The result: you compile once on a 5.15 machine and the program loads correctly on any kernel with BTF support (5.2+ has BTF; modern distros backport it).

The idiom in BPF C code is BPF_CORE_READ(task, pid) instead of task->pid. The macro expands to a __builtin_preserve_access_index() call that Clang marks as needing relocation.

vmlinux.h is the companion tool: bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h generates a single header with all kernel struct definitions derived from the running kernel's BTF. You include this instead of kernel headers — no more #include <linux/sched.h> fragility.

/* CO-RE example: read task->comm portably */
#include "vmlinux.h"          /* generated from target kernel's BTF */
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

SEC("tracepoint/sched/sched_switch")
int trace_sched_switch(struct trace_event_raw_sched_switch *ctx)
{
    /* BPF_CORE_READ: field offset resolved at load time via BTF */
    struct task_struct *prev =
        (struct task_struct *)BPF_CORE_READ(ctx, prev);

    char comm[16];
    BPF_CORE_READ_STR_INTO(&comm, prev, comm);

    bpf_printk("switched away from: %s\n", comm);
    return 0;
}

char _license[] SEC("license") = "GPL";

BPF programs cannot call arbitrary kernel functions — that would bypass the safety guarantees. Instead, they call BPF helper functions: a curated, versioned set of kernel functions with stable signatures, declared in include/uapi/linux/bpf.h. As of kernel 6.x, there are over 200 helpers.

Helpers by category:

Map operations: bpf_map_lookup_elem, bpf_map_update_elem, bpf_map_delete_elem, bpf_map_push_elem, bpf_map_pop_elem.

Packet manipulation: bpf_skb_load_bytes, bpf_skb_store_bytes, bpf_l3_csum_replace, bpf_l4_csum_replace, bpf_skb_adjust_room (add/remove header bytes), bpf_redirect (redirect packet to another interface).

Tracing/debugging: bpf_printk (prints to /sys/kernel/debug/tracing/trace_pipe; slow, for debugging only), bpf_perf_event_output (send data to perf ring buffer), bpf_ringbuf_output (send to BPF ring buffer, preferred).

Timing: bpf_ktime_get_ns (nanoseconds since boot, fast), bpf_ktime_get_tai_ns (TAI wall clock).

Process/task info: bpf_get_current_pid_tgid (returns (tgid << 32) | pid as u64), bpf_get_current_comm (fills a buffer with the task name), bpf_get_current_task (returns a pointer to struct task_struct, usable with CO-RE).

Spin locks: bpf_spin_lock / bpf_spin_unlock — for protecting a value in a map against concurrent access across CPUs (since BPF programs can run in parallel on different cores).

BPF-to-BPF calls: since kernel 4.16, a BPF program can call a subfunction defined in the same program. The verifier inlines or call-frames it; the JIT emits a real call. This allows code reuse without tail calls.

kfuncs (kernel functions for BPF, 5.13+): a newer extensibility mechanism where kernel subsystems can explicitly export ordinary kernel functions as callable from BPF, with full BTF type checking. This is rapidly replacing the old helper interface for new functionality — e.g., bpf_kfunc_call_test_acquire for testing, bpf_list_push_front for BPF-managed linked lists.

/* Using bpf_get_current_pid_tgid + ring buffer to emit events */
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

struct event {
    __u32 pid;
    char  comm[16];
    __u64 ts_ns;
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024); /* 256 KB ring */
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_execve")
int trace_exec(void *ctx)
{
    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e) return 0;

    e->pid   = bpf_get_current_pid_tgid() >> 32;
    e->ts_ns = bpf_ktime_get_ns();
    bpf_get_current_comm(e->comm, sizeof(e->comm));

    bpf_ringbuf_submit(e, 0); /* zero-copy: pointer into ring, no memcpy */
    return 0;
}
char _license[] SEC("license") = "GPL";

Understanding eBPF purely through code examples misses the larger story. eBPF has reshaped the Linux ecosystem in three domains:

Networking: Cilium (https://cilium.io) replaced iptables-based Kubernetes networking with a pure-BPF dataplane. iptables is O(n) in the number of rules; BPF hash maps are O(1). At 10,000 pods, iptables processing latency is measured in milliseconds; Cilium's BPF dataplane is measured in microseconds. Meta's Katran does L4 load balancing entirely in XDP, handling tens of millions of packets per second per machine. Cloudflare uses XDP to absorb Terabit-scale DDoS attacks.

Observability: Every major cloud vendor — Google, Meta, Netflix, LinkedIn — runs production profiling via BPF. The profile:99hz bpftrace probe turns every CPU into a continuous flame-graph source with near-zero overhead (the cost is the BPF program execution on a timer tick, not a heavyweight system call). Continuous profiling tools like Parca and Pyroscope are built on this. bpftool prog profile can measure BPF program CPU cycles, instruction counts, and cache misses using hardware performance counters.

Security: KRSI (Kernel Runtime Security Instrumentation) allows security policy to be expressed in BPF LSM programs and updated without rebooting. Falco, a popular cloud-native threat detection tool, recently migrated its kernel driver to BPF. Google's gVisor uses BPF for system call filtering. The Landlock LSM (part of the mainline kernel since 5.13) provides sandboxing without root, using BPF-adjacent concepts.

The developer experience flywheel: The combination of bpftrace (fast iteration), libbpf + skeletons (production use), CO-RE (portability), and BTF (introspection) means the barrier to writing production BPF programs has dropped dramatically. The bpftool utility can list all running BPF programs on a system (bpftool prog list), dump their bytecode or JIT output, inspect map contents, and generate skeletons — it is like ps + gdb + a compiler driver for the BPF world.

From a kernel contribution standpoint, eBPF is a young, fast-moving subsystem. The BPF mailing list (bpf@vger.kernel.org) is one of the most active on LKML. Good first contributions: adding test cases to tools/testing/selftests/bpf/, fixing documentation, or implementing a new kfunc for an existing subsystem. The maintainers (Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko) are known for thorough, technical, and welcoming code reviews.