Scheduler Extensibility: sched_ext & BPF Schedulers

The Completely Fair Scheduler (CFS), introduced in Linux 2.6.23, is a remarkable engineering achievement. It uses a red-black tree keyed on vruntime (virtual runtime) to pick the task that has been most starved of CPU time, producing fair, low-latency scheduling across a huge diversity of workloads. But "fair" is not always what you want.

Consider three completely different use cases:

Gaming: A game has one high-priority render thread that must hit a frame deadline every 16.6ms (60fps). It also has dozens of worker threads, audio threads, and background loaders. CFS treats them all as roughly equal unless you manually set nice values or use SCHED_FIFO. The render thread can get bumped by a low-priority worker at the worst moment, causing a visible stutter. Game engine companies have entire teams of engineers whose job is working around the scheduler.

Datacenter batch computing: A server running thousands of short-lived tasks (think Hadoop map tasks, each lasting <1 second) spends enormous time in context-switch overhead. CFS's default tick-based preemption and wakeup heuristics were tuned for interactive desktop use, not for "run this 10ms task and immediately run the next one from a queue."

Heterogeneous CPUs: Modern ARM chips (like Apple Silicon and Qualcomm Snapdragon) and Intel's P/E-core chips have asymmetric cores. A task that is CPU-bound on an efficiency core should stay there, while a latency-sensitive task should migrate to a performance core immediately on wakeup. CFS knows about this via Energy-Aware Scheduling (EAS), but the heuristics are baked in.

In all three cases, the application or fleet operator has information the kernel does not: which tasks are on the critical path, what the memory access patterns look like, what the deadline is. The problem is that this information cannot be communicated to CFS in a useful way. sched_ext solves this by letting you write the scheduling policy that uses your information.

# Check if your kernel has sched_ext support (v6.12+)
uname -r
grep CONFIG_SCHED_CLASS_EXT /boot/config-$(uname -r)

# List available SCX schedulers (if scx-tools package installed)
ls /usr/lib/scx/ 2>/dev/null || echo "Install scx-tools for reference schedulers"

The Linux kernel has a concept of scheduler classes, a priority-ordered linked list of objects each implementing a set of function pointers (pick_next_task, enqueue_task, dequeue_task, etc.). The classes are, in order of priority: stop_sched_class (used internally for CPU migration), dl_sched_class (deadline scheduling), rt_sched_class (real-time), fair_sched_class (CFS), and idle_sched_class.

sched_ext inserts a new class, ext_sched_class, between fair and idle. When a task opts in to SCX (either explicitly, or when a global flag is set), it moves out of the CFS run queue and into the SCX subsystem. From that point, pick_next_task_scx() is called whenever the CPU needs a new task from SCX, and it delegates to a BPF program via the ops dispatch points.

The core SCX abstractions:

• DSQ (Dispatch Queue): A queue of tasks ready to run. Every CPU has a built-in local DSQ (SCX_DSQ_LOCAL). There is also a global FIFO DSQ (SCX_DSQ_GLOBAL). You can create custom DSQs with scx_bpf_create_dsq(). Your BPF program enqueues tasks into DSQs and the kernel dispatches from them.

• ops.enqueue: Called when a task becomes runnable (wakes up or is preempted). Your BPF program decides which DSQ to put it in.

• ops.dispatch: Called when a CPU's local DSQ is empty and it needs more work. Your BPF program can move tasks from a shared DSQ to the local DSQ, implementing any load-balancing policy you want.

• ops.select_cpu: An optimization hook called before enqueue that can directly wake up a target CPU, bypassing the generic wakeup path.

• ops.running / ops.stopping: Called when a task starts and stops running on a CPU. Used to update statistics, time slices, etc.

The beauty of this design is that the fast path (dispatch from local DSQ to CPU) is implemented entirely in kernel C code that is always correct and efficient. Your BPF program only needs to make policy decisions about which DSQ to use and in what order.

// Conceptual view of the dispatch loop (kernel C, simplified)
// From kernel/sched/ext.c
static struct task_struct *pick_next_task_scx(struct rq *rq) {
    struct scx_rq *scx_rq = &rq->scx;
    struct task_struct *p;

    // First: drain the local DSQ (no BPF call needed, pure kernel)
    p = first_local_dsq_task(scx_rq);
    if (p) return p;

    // Local DSQ empty: call ops.dispatch to let BPF refill it
    if (scx_ops.dispatch)
        SCX_CALL_OP(SCX_KF_DISPATCH, dispatch, cpu_of(rq), NULL);

    // Try local DSQ again after BPF dispatch
    p = first_local_dsq_task(scx_rq);
    if (p) return p;

    // Fall through to idle or other classes
    return NULL;
}

The most important question about sched_ext is: what stops a buggy BPF scheduler from panicking the kernel? The answer is a three-layer safety net.

Layer 1: The BPF Verifier. Before a BPF program is ever loaded into the kernel, the verifier statically analyzes it. It performs abstract interpretation: it simulates the execution of every possible path through the program, tracking the type and bounds of every register and memory access. It verifies that:

• The program terminates (no unbounded loops — though bounded loops are allowed in modern BPF)
• All memory accesses are in-bounds and of the correct type
• The program only calls helper functions it is permitted to call
• Pointer arithmetic does not escape safe regions

For sched_ext, the verifier also checks that BPF programs called in specific contexts (e.g., ops.enqueue, which is called with the run queue lock held) only call helpers that are safe to use in that context. This is enforced via a kfunc call graph that tags each kfunc with a bitmask of allowed calling contexts.

Layer 2: SCX Watchdog. Even a verifier-approved program can have logical bugs: it might enqueue tasks to a DSQ that never gets dispatched, causing CPUs to go idle. SCX runs a watchdog timer (default: 5 seconds) that checks if any runnable SCX task has been waiting too long. If so, it triggers a graceful error path that disables the BPF scheduler and falls back to the next scheduler class. The system does not panic; it just stops using your scheduler.

Layer 3: ops.init_task / ops.exit_task. These hooks let your scheduler do per-task setup and teardown. If ops.init_task returns an error (e.g., memory allocation failed), the kernel rejects the task opt-in cleanly. If ops implementation exits uncleanly, the kernel can evict the SCX scheduler and return all tasks to CFS.

The result is a remarkable safety property: a buggy sched_ext scheduler can cause performance degradation or even temporary CPU stalls, but it cannot cause a kernel panic or data corruption. This is qualitatively different from a traditional scheduler bug.

// The BPF verifier rejects this program:
// - unbounded loop (before bounded-loop support)
// - out-of-bounds array access
SEC("struct_ops/enqueue")
void BPF_PROG(bad_scheduler_enqueue, struct task_struct *p, u64 enq_flags) {
    int i;
    // ERROR: verifier sees this as potentially infinite
    // (without proving bounds)
    for (i = 0; ; i++) {
        if (p->pid == i) break;
    }
    scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}

// The verifier ACCEPTS this:
SEC("struct_ops/enqueue")
void BPF_PROG(good_enqueue, struct task_struct *p, u64 enq_flags) {
    // Bounded: verifier can prove termination
    scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}

The canonical starting point for learning sched_ext is scx_simple, the minimal reference scheduler included in the kernel's tools/sched_ext/ directory and also in the upstream sched-ext/scx GitHub repository. It implements a global FIFO scheduler in about 80 lines of BPF C.

A BPF scheduler is structured as a struct_ops BPF program. The sched_ext kernel module exposes a struct sched_ext_ops type, and your BPF program provides implementations of its fields. The user-space loader (written in C using libbpf) opens the BPF object, instantiates the struct_ops map, and calls bpf_map_update_elem to register it with the kernel.

The key insight about scx_simple is what it does not implement: it uses the global FIFO DSQ (SCX_DSQ_GLOBAL) for everything, meaning ops.dispatch is essentially free — the kernel's built-in global-to-local dispatch logic handles it. The enqueue hook just calls scx_bpf_dispatch() with SCX_DSQ_GLOBAL and the default time slice. This is not a good scheduler for real workloads (global FIFO has poor cache locality), but it is a perfect pedagogical base.

After you load scx_simple, all normal tasks (those not using SCHED_FIFO, SCHED_DEADLINE, etc.) are scheduled by it. CFS is completely bypassed for those tasks. When you unload the scheduler (Ctrl-C or scx_bpf_exit()), all tasks seamlessly return to CFS.

// scx_simple.bpf.c — a minimal global-FIFO BPF scheduler
// (condensed from tools/sched_ext/scx_simple.bpf.c)
#include <scx/common.bpf.h>

char _license[] SEC("license") = "GPL";

// Called when a task becomes runnable.
// We just send everything to the global FIFO DSQ.
SEC("struct_ops/enqueue")
void BPF_PROG(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
    scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}

// ops.dispatch is not implemented — the kernel's default
// global-DSQ-to-local-DSQ dispatch logic handles it.

// Declare the struct_ops implementation
SEC(".struct_ops.link")
struct sched_ext_ops simple_ops = {
    .enqueue   = (void *)simple_enqueue,
    .name      = "simple",
    // .flags = SCX_OPS_ENQ_LAST | SCX_OPS_KEEP_BUILTIN_IDLE,
};

// ---- User-space loader (simplified) ----
// int main() {
//     struct scx_simple *skel = scx_simple__open();
//     scx_simple__load(skel);
//     struct bpf_link *link = bpf_map__attach_struct_ops(
//                                 skel->maps.simple_ops);
//     // scheduler is now ACTIVE
//     pause(); // run until Ctrl-C
//     bpf_link__destroy(link); // back to CFS
// }

A global FIFO scheduler is simple but terrible for multi-core performance. The problem is cache affinity: if task A last ran on CPU 3, its data is warm in CPU 3's L1/L2 caches. A global FIFO might dispatch it to CPU 7 next, causing cold-cache misses. Good schedulers prefer to keep tasks on the same CPU (CPU affinity) while still load-balancing so no CPU sits idle while others are overloaded.

The classic solution is per-CPU run queues with work stealing: each CPU has its own queue, tasks are preferentially re-enqueued to the CPU they last ran on, but idle CPUs can steal tasks from overloaded CPUs.

sched_ext supports this naturally:

1. Create one DSQ per CPU: scx_bpf_create_dsq(cpu_id, -1) (the -1 means no cgroup association)
2. In ops.enqueue, dispatch the task to SCX_DSQ_LOCAL_ON | cpu_id (the DSQ for a specific CPU), picking the CPU based on affinity
3. In ops.dispatch, if the local DSQ is empty, call scx_bpf_consume() on neighboring CPUs' DSQs (work stealing)

scx_rusty (written in Rust using Rust's libbpf-rs bindings) is a production-quality scheduler from Meta that implements exactly this pattern, with NUMA-awareness, load-tracking via EWMA, and a configurable work-stealing radius. It is used in production on Meta's fleet.

The architectural lesson here is that sched_ext separates mechanism (DSQs, dispatch, preemption) from policy (which DSQ, when to steal, how much time slice to give). The BPF program only needs to express policy, and the kernel's mechanism is always correct.

// Per-CPU DSQ work-stealing pattern (illustrative)
#include <scx/common.bpf.h>

// One DSQ per CPU, created at ops.init
// dsq_id[cpu] = cpu (we use cpu number as DSQ id)

SEC("struct_ops/enqueue")
void BPF_PROG(ws_enqueue, struct task_struct *p, u64 enq_flags)
{
    s32 cpu = scx_bpf_task_cpu(p); // last CPU the task ran on
    // Try to keep task on same CPU for cache affinity
    scx_bpf_dispatch(p, cpu, SCX_SLICE_DFL, enq_flags);
}

SEC("struct_ops/dispatch")
void BPF_PROG(ws_dispatch, s32 cpu, struct task_struct *prev)
{
    s32 victim;
    // Local DSQ is empty — try to steal from neighbors
    bpf_for(victim, 0, nr_cpu_ids) {
        if (victim == cpu) continue;
        // Move up to 1 task from victim's DSQ to our local DSQ
        if (scx_bpf_dsq_move_to_local(victim))
            return; // got one, done
    }
}

A crucial concept in any scheduler is preemption: how does the kernel take the CPU away from a running task? In CFS, tasks are preempted either when a higher-priority task wakes up (wakeup preemption) or when the scheduler tick fires and the task has overrun its time slice.

In sched_ext, time slices work differently. When your ops.enqueue dispatches a task, you specify a time slice (the third argument to scx_bpf_dispatch()). SCX_SLICE_DFL is the default (around 20ms). You can use any value in nanoseconds. When the task has used its time slice, the kernel calls ops.enqueue again with the SCX_ENQ_PREEMPT flag, and your scheduler re-enqueues the task (possibly to a different DSQ, e.g., a lower-priority one if you want priority aging).

For wakeup preemption, sched_ext has ops.select_cpu and the SCX_ENQ_PREEMPT path. If a high-priority task wakes up and you want it to run immediately, you can use scx_bpf_kick_cpu() to send an inter-processor interrupt (IPI) to a target CPU, telling it to reschedule. This is the same mechanism CFS uses for wakeup preemption.

A subtle but important flag is SCX_OPS_ENQ_LAST. By default, when a task is the only runnable task on a CPU, sched_ext does not call ops.enqueue — it just lets the task run. With SCX_OPS_ENQ_LAST, your scheduler is called even in this case, giving you the chance to inspect the task (useful for accounting or tracing).

The tick itself is configurable. You can set ops.flags |= SCX_OPS_SWITCH_PARTIAL to make only some tasks go through SCX (leaving others in CFS), which is useful for experimentation.

// Deadline-aware enqueue: give short slice to interactive tasks,
// long slice to CPU-bound tasks
SEC("struct_ops/enqueue")
void BPF_PROG(deadline_enqueue, struct task_struct *p, u64 enq_flags)
{
    u64 slice;
    u64 avg_runtime;

    // scx_bpf_task_cgroup and per-task storage can give us hints;
    // here we use a simple heuristic: tasks that voluntarily yield
    // (enq_flags has SCX_ENQ_WAKEUP set) are likely interactive
    if (enq_flags & SCX_ENQ_WAKEUP) {
        slice = 1 * 1000 * 1000; // 1ms — interactive, short slice
    } else {
        slice = 20 * 1000 * 1000; // 20ms — CPU-bound, long slice
    }

    // Dispatch to global DSQ with computed slice
    scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, slice, enq_flags);
}

A scheduler needs to track state: how long has each task been waiting? What is its priority? Has it exceeded its budget? In a kernel scheduler, you would add fields to struct task_struct. In BPF, you cannot modify struct task_struct directly — but you have BPF maps.

sched_ext provides two mechanisms for per-task storage:

1. struct_ops storage maps: You can declare a BPF per-task storage map of type BPF_MAP_TYPE_TASK_STORAGE. This is a hash map keyed by task pointer, allocated per-task at ops.init_task time and freed at ops.exit_task. It gives you a fixed-size struct of your own design for every SCX task.

2. scx_bpf_task_running() and task fields: Many fields of struct task_struct are readable directly from BPF (the verifier knows their types via BTF — BPF Type Format, which we discuss next). You can read p->pid, p->tgid, p->prio, p->se.vruntime, and many others directly.

BTF (BPF Type Format) is the mechanism that makes this safe. The kernel exports its type information (every struct, its fields, their types and offsets) in a compact format embedded in the kernel image. The BPF verifier uses BTF to verify that a field access like p->se.vruntime is accessing a known field of the known type at the correct offset, with the correct size. No arbitrary memory access is possible; every access is name-checked against the actual kernel type definition.

// Per-task storage for a custom scheduler
#include <scx/common.bpf.h>

// Define what we want to track per task
struct task_ctx {
    u64 last_run_ns;   // timestamp of last run
    u64 total_run_ns;  // cumulative CPU time
    u32 slice_used;    // number of slices used (for aging)
};

// Declare the per-task storage map
struct {
    __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
    __uint(map_flags, BPF_F_NO_PREALLOC);
    __type(key, int);
    __type(value, struct task_ctx);
} task_storage SEC(".maps");

// In ops.running: update last-run timestamp
SEC("struct_ops/running")
void BPF_PROG(my_running, struct task_struct *p)
{
    struct task_ctx *ctx = bpf_task_storage_get(
        &task_storage, p, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE);
    if (ctx)
        ctx->last_run_ns = bpf_ktime_get_ns();
}

// In ops.stopping: accumulate CPU time
SEC("struct_ops/stopping")
void BPF_PROG(my_stopping, struct task_struct *p, bool runnable)
{
    struct task_ctx *ctx = bpf_task_storage_get(
        &task_storage, p, NULL, 0);
    if (ctx) {
        u64 now = bpf_ktime_get_ns();
        ctx->total_run_ns += now - ctx->last_run_ns;
        ctx->slice_used++;
    }
}

Gaming is the killer use case that initially drove much of the community enthusiasm for sched_ext. The problem: a game running on Linux (especially through Proton/Wine) has complex thread interdependencies that the kernel cannot see. The main render thread depends on worker threads to finish their tasks before it can produce a frame. If CFS gives the render thread its time slice at 14ms and a worker thread at 16ms, the frame is late. If both got time at 8ms and 10ms, the frame could have been on time.

scx_lavd (Latency-Criticality Aware Virtual Deadline) is a scheduler designed by Changwoo Min at Igalia specifically for gaming and mixed interactive/batch workloads. Its key ideas:

1. Latency criticality metric: LAVD estimates how latency-critical a task is by observing its wakeup patterns. Tasks that wake up frequently (short sleep intervals) tend to be interactive; tasks that run long before blocking are CPU-bound. LAVD computes a per-task latency_criticality score using an EWMA (exponentially weighted moving average) of inter-wakeup intervals.

2. Virtual deadline: Each task is assigned a virtual deadline based on its latency criticality. More critical tasks get nearer deadlines and thus higher priority. This is similar in spirit to EDF (Earliest Deadline First) scheduling.

3. Preemption for newly critical tasks: When a high-criticality task wakes up, LAVD calls scx_bpf_kick_cpu() to preempt the currently running task on the target CPU, even if that task has time remaining in its slice. This is the key to achieving low latency for the render thread.

Experimental results showed scx_lavd reducing frame time variance in games like Shadow of the Tomb Raider by 30-50% compared to CFS on some hardware configurations. It is now included in the upstream sched-ext/scx repository and shipped by distributions like Fedora and Bazzite.

The broader lesson is that scheduling is a domain-specific problem. A scheduler that is great for games may be terrible for a database server. sched_ext is the first Linux mechanism that makes it practical to have per-use-case schedulers without maintaining a fleet of custom kernels.

// Conceptual sketch of LAVD's criticality-based dispatch
// (not the real scx_lavd, but illustrates the core idea)

// Per-task context (stored in BPF task storage)
struct lavd_task_ctx {
    u64 last_wakeup_ns;
    u64 avg_wakeup_interval_ns;  // EWMA
    u64 virtual_deadline;        // assigned deadline
};

static u64 compute_criticality(struct lavd_task_ctx *ctx) {
    // Short avg interval = wakes up often = latency-critical
    // Maps to a small deadline (higher priority)
    if (ctx->avg_wakeup_interval_ns < 1000000ULL) // <1ms
        return 0; // highest priority (earliest deadline)
    return ctx->avg_wakeup_interval_ns; // longer = lower priority
}

SEC("struct_ops/enqueue")
void BPF_PROG(lavd_enqueue, struct task_struct *p, u64 enq_flags)
{
    struct lavd_task_ctx *ctx = /* get from task storage */ NULL;
    if (!ctx) {
        scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
        return;
    }

    u64 vdeadline = bpf_ktime_get_ns() + compute_criticality(ctx);
    ctx->virtual_deadline = vdeadline;

    // Dispatch to a deadline-ordered DSQ
    // (scx_bpf_dispatch_vtime uses the vtime field for ordering)
    scx_bpf_dispatch_vtime(p, LAVD_DSQ, SCX_SLICE_DFL,
                           vdeadline, enq_flags);
}

In a data center, the concerns are almost the opposite of gaming: you care about throughput and energy efficiency, not individual frame latencies. The key technique is core compaction (also called core parking): instead of spreading tasks evenly across all cores (which keeps all cores active and burning power), pack tasks onto the fewest cores necessary, leaving the rest completely idle so they can enter deep C-states and save power.

CFS is notoriously bad at core compaction. Its load balancer actively tries to spread tasks for fairness, constantly migrating tasks to underloaded cores. This fights directly against compaction.

scx_nest is a scheduler from Google that implements a two-tier core model:

• Primary cores: A small set of CPUs that always have tasks. Tasks migrate here eagerly.
• Reserve cores: CPUs that are only activated when the primary set is overloaded.

The BPF scheduler tracks which CPUs are in the primary set using a BPF_MAP_TYPE_CPUMASK. Tasks are preferentially dispatched to primary CPUs. A periodic timer (using bpf_timer) checks if primary CPUs are overloaded and promotes reserve CPUs as needed, or demotes them back when load drops.

Google reported that scx_nest reduced energy consumption by ~10% in some workloads compared to CFS on Google's fleet, while maintaining similar throughput.

scx_bpfland is another approach, targeting mixed interactive + batch workloads on desktop/server. It classifies tasks into interactive (using is_interactive heuristic based on average run time) and regular buckets, serving interactive tasks first.

The datacenter story also includes heterogeneous CPU management. On a server with Intel Efficiency cores and Performance cores, an SCX scheduler can explicitly assign batch tasks to E-cores and wake up P-cores only for latency-sensitive tasks, implementing a policy that AMDs and Intel's own kernel patches try to bake in as generic heuristics.

// scx_nest core compaction concept
// Primary set tracking via BPF cpumask

// Map to track which CPUs are "primary"
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, u32);
    __type(value, struct bpf_cpumask *);
} primary_cpumask SEC(".maps");

SEC("struct_ops/select_cpu")
s32 BPF_PROG(nest_select_cpu, struct task_struct *p,
             s32 prev_cpu, u64 wake_flags)
{
    struct bpf_cpumask *primary;
    s32 cpu;
    u32 key = 0;

    primary = bpf_map_lookup_elem(&primary_cpumask, &key);
    if (!primary)
        return prev_cpu;

    // Try to find an idle CPU in the primary set first
    cpu = scx_bpf_pick_idle_cpu(
              (const struct cpumask *)primary, 0);
    if (cpu >= 0) {
        scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu,
                         SCX_SLICE_DFL, wake_flags);
        return cpu;
    }

    // Primary CPUs busy — fall back to any idle CPU
    // (reserve CPU will be promoted by the timer)
    return scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
}

sched_ext is built on top of a more general BPF feature called struct_ops, which is worth understanding in its own right because it will appear again in future BPF-extensible kernel subsystems.

The idea: the kernel defines a C struct of function pointers (like struct sched_ext_ops). A BPF program can provide implementations of those function pointers. The kernel's BPF infrastructure handles:

1. Registering the BPF program's implementations for each field
2. Calling the BPF programs at the right times (with the right arguments)
3. Ensuring the BPF verifier checks each implementation in the context of how it will be called

Before struct_ops (and before sched_ext), BPF had cgroup hooks and tracing hooks, but these were read-only (you could observe but not change behavior). struct_ops is the first BPF mechanism for replacing kernel behavior, not just observing it.

The struct sched_ext_ops type has about 30 function pointer fields. Most are optional — a minimal scheduler only needs enqueue. This makes the API progressive: start with enqueue, learn the behavior, then add dispatch, then select_cpu, then running/stopping for accounting.

The user-space side of struct_ops uses libbpf. The BPF skeleton (auto-generated by bpftool gen skeleton) gives you a C struct that mirrors your BPF object, so you can set ops fields in C:

skel->maps.my_ops.ops.name = "my_scheduler";

Then you call bpf_map__attach_struct_ops(skel->maps.my_ops) to activate it.

// The full struct sched_ext_ops interface (selected fields)
// From include/linux/sched/ext.h
struct sched_ext_ops {
    // Core scheduling ops
    s32  (*select_cpu)(struct task_struct *p, s32 prev_cpu, u64 wake_flags);
    void (*enqueue)(struct task_struct *p, u64 enq_flags);
    void (*dequeue)(struct task_struct *p, u64 deq_flags);
    void (*dispatch)(s32 cpu, struct task_struct *prev);

    // Task lifecycle
    void (*runnable)(struct task_struct *p, u64 enq_flags);
    void (*running)(struct task_struct *p);
    void (*stopping)(struct task_struct *p, bool runnable);
    void (*quiescent)(struct task_struct *p, u64 deq_flags);

    // CPU lifecycle
    void (*cpu_online)(s32 cpu);
    void (*cpu_offline)(s32 cpu);

    // Initialization and exit
    s32  (*init)(void);
    void (*exit)(struct scx_exit_info *info);
    s32  (*init_task)(struct task_struct *p,
                      struct scx_init_task_args *args);
    void (*exit_task)(struct task_struct *p,
                      struct scx_exit_task_args *args);

    // Configuration
    u32  flags;    // SCX_OPS_* flags
    u32  timeout_ms;
    char name[SCX_OPS_NAME_LEN];
};

The practical workflow for developing a BPF scheduler has several components. Let us walk through the full development cycle.

Prerequisites:

• Linux kernel 6.12+ with CONFIG_SCHED_CLASS_EXT=y and CONFIG_BPF_SYSCALL=y
• clang and llvm for BPF compilation
• libbpf development headers
• bpftool for skeleton generation

Step 1: Get the scx repository. The canonical source of reference schedulers and the build system is https://github.com/sched-ext/scx. Clone it and build with meson:

git clone https://github.com/sched-ext/scx
cd scx && meson setup build && cd build && ninja

Step 2: Load a reference scheduler. Start with scx_simple:

sudo ./scheds/c/scx_simple

All non-RT tasks are now scheduled by your BPF program. Use htop, perf sched, or bpftool prog list to observe.

Step 3: Instrument with bpf_printk. During development, bpf_printk() writes to /sys/kernel/debug/tracing/trace_pipe, giving you printf-style debugging from inside the kernel.

Step 4: Use scxtop. The scxtop tool in the scx repo is a TUI (like htop) that shows per-DSQ statistics, dispatch latencies, and task migration rates in real time. Essential for understanding what your scheduler is actually doing.

Step 5: Benchmarking. Use schbench (scheduling benchmark) to measure scheduler latency and throughput. Compare your scheduler to CFS baseline:

schbench -m 2 -t 16 -r 30  # 2 message-passing threads, 16 worker threads, 30 sec

Step 6: Stress testing. The scx_test tool runs your scheduler under artificial load and deliberately tries to trigger the watchdog. A good scheduler should survive this.

# Full build-and-test workflow

# 1. Verify kernel support
grep CONFIG_SCHED_CLASS_EXT /boot/config-$(uname -r)
# Should show: CONFIG_SCHED_CLASS_EXT=y

# 2. Build scx schedulers
git clone https://github.com/sched-ext/scx
cd scx
meson setup build --prefix=/usr
cd build && ninja

# 3. Run the minimal scheduler
sudo ./scheds/c/scx_simple -v  # -v for verbose

# 4. In another terminal: watch trace output
sudo cat /sys/kernel/debug/tracing/trace_pipe

# 5. Check loaded BPF programs
bpftool prog list | grep sched

# 6. Benchmark against CFS
# (first, stop scx_simple: Ctrl-C restores CFS)
schbench -m 2 -t $(nproc) -r 30 2>&1 | tee cfs_baseline.txt
sudo ./scheds/c/scx_simple &
schbench -m 2 -t $(nproc) -r 30 2>&1 | tee scx_simple.txt
kill %1  # stop scx_simple
diff cfs_baseline.txt scx_simple.txt

As of Linux 6.12, sched_ext is mainline. What does its future look like?

Distribution adoption: Fedora 40+ ships scx-scheds as an optional package. Bazzite (a gaming-focused Fedora spin) ships with scx_lavd as the default scheduler. CachyOS ships multiple SCX schedulers selectable via a GUI. This is the fastest adoption curve of any new kernel scheduling mechanism since CFS itself.

Upstream scheduler development: Several traditional scheduler improvements are being prototyped in sched_ext first. The EEVDF (Earliest Eligible Virtual Deadline First) scheduler, which replaced CFS's internal algorithm in 6.6, was influenced by ideas more easily experimented with in pluggable form. Future work on server-class NUMA-aware scheduling is increasingly done in SCX first, then potentially upstreamed as CFS patches.

Rust BPF schedulers: The scx repository includes a Rust library (scx_utils) that wraps libbpf-rs, making it possible to write the user-space half of a scheduler in Rust. scx_rusty and scx_layered are production-ready Rust schedulers. This is significant: the ergonomics of Rust (sum types, the borrow checker, cargo package management) make complex scheduling logic much easier to write correctly than C.

Automated scheduler selection: Research projects (including work at Google) explore using ML models to select or tune scheduler parameters based on observed workload characteristics. With sched_ext, the feedback loop is tight: change the BPF program, reload it at runtime, measure, repeat — no reboots.

Beyond scheduling: struct_ops is now being used in other kernel subsystems (TCP congestion control via BPF, HID driver customization). sched_ext proved the architecture works. Expect to see BPF-based pluggability in memory reclaim, I/O schedulers, and network QoS in future kernel versions.

The philosophical shift is profound: the kernel is moving from a monolithic policy engine to a verified, extensible mechanism substrate. sched_ext is the most visible symbol of that shift today.

# Selecting a scheduler at runtime (systemd-based)
# The scx.service unit switches schedulers without a reboot

# Install scx-scheds (Fedora)
sudo dnf install scx-scheds

# Enable and start the SCX service with scx_lavd for gaming
sudo systemctl edit scx --force
# Add:
# [Service]
# Environment=SCX_SCHEDULER=scx_lavd

sudo systemctl enable --now scx
systemctl status scx

# Switch to scx_nest for server/power-saving workload
sudo systemctl stop scx
sudo SCX_SCHEDULER=scx_nest systemctl start scx

# Check which scheduler is active
cat /sys/kernel/sched_ext/root/ops   # shows active ops name
cat /sys/kernel/sched_ext/state      # shows: enabled/disabled