The Linux Scheduler: CFS, EEVDF & Scheduling Classes

Before diving into any algorithm, ask: what are we actually trying to achieve? The scheduler must simultaneously satisfy several conflicting goals:

1. Fairness — processes with equal priority should get equal CPU time over any long window.
2. Throughput — the CPU should rarely be idle when there is work to do.
3. Low latency — interactive tasks (terminals, audio, games) must respond within milliseconds.
4. Real-time guarantees — some tasks have hard deadlines (a kernel thread servicing an interrupt-driven device).
5. Energy efficiency — on a laptop, idle CPUs should sleep.

Linux resolves the tension between these goals with a hierarchy of scheduling classes. Each class implements a standard interface (a sched_class struct of function pointers), and the scheduler simply walks the hierarchy from highest to lowest priority, asking each class "do you have a runnable task?" The first class to say yes wins.

The five classes, in order of absolute priority:

The critical insight is that a SCHED_DEADLINE task will always preempt any SCHED_NORMAL task, no matter how long the normal task has been waiting. This is not unfairness — it is deliberate policy. Real-time guarantees are meaningless if a video codec can be preempted by a spreadsheet calculation.

The sched_class interface in kernel/sched/sched.h defines hooks like enqueue_task, dequeue_task, pick_next_task, task_tick, and check_preempt_curr. Each scheduling class fills in these hooks with its own logic. This is Linux's version of polymorphism — written in C.

/* From kernel/sched/sched.h — the sched_class interface (simplified) */
struct sched_class {
    /* Called when a task becomes runnable (wakes up or is created) */
    void (*enqueue_task)(struct rq *rq, struct task_struct *p, int flags);

    /* Called when a task stops being runnable (blocks, exits) */
    void (*dequeue_task)(struct rq *rq, struct task_struct *p, int flags);

    /* Select the next task to run on this CPU */
    struct task_struct *(*pick_next_task)(struct rq *rq);

    /* Called on every timer tick for the current task */
    void (*task_tick)(struct rq *rq, struct task_struct *p, int queued);

    /* Check if the current task should be preempted */
    void (*check_preempt_curr)(struct rq *rq, struct task_struct *p, int flags);

    /* Linked list: next class to check if this one has no runnable tasks */
    const struct sched_class *next;
};

/* The class hierarchy is declared in kernel/sched/core.c:
 * stop_sched_class -> dl_sched_class -> rt_sched_class
 *   -> fair_sched_class -> idle_sched_class
 */

Each CPU has exactly one runqueue (struct rq in kernel/sched/sched.h). Think of the runqueue as the CPU's "inbox" — it holds all the tasks that are ready to run on that CPU and all the per-CPU scheduler state.

The struct rq is large and contains sub-runqueues for each scheduling class:

• struct cfs_rq cfs — the CFS/EEVDF red-black tree
• struct rt_rq rt — the RT priority array
• struct dl_rq dl — the deadline queue

Other important fields include:

• curr — pointer to the currently running task
• nr_running — total number of runnable tasks
• clock and clock_task — a monotonic clock used to track CPU time; updated on every context switch and timer tick
• lock — a spinlock protecting the entire runqueue (this is one of the most contended locks in the kernel; reducing contention here has been a major engineering focus)

Because each CPU has its own runqueue, most scheduling operations are lock-free on the fast path — a CPU picks its next task from its own runqueue without needing to communicate with any other CPU. Load balancing (moving tasks between CPUs) does require cross-CPU coordination, but it happens less frequently.

One subtle but important field is rq->clock_task. The scheduler needs to know how much CPU time a task actually consumed, but a task can be interrupted by hardware interrupts, softirqs, and other non-task work. clock_task is only advanced when the CPU is actually running a task (not handling interrupts), so CFS's virtual runtime accounting is based on actual task execution time, not wall-clock time.

/* Simplified view of struct rq (kernel/sched/sched.h) */
struct rq {
    raw_spinlock_t      lock;           /* protects this runqueue */
    unsigned int        nr_running;     /* number of runnable tasks */
    u64                 clock;          /* monotonic clock, ns */
    u64                 clock_task;     /* clock excluding interrupt time */

    struct task_struct  *curr;          /* currently running task */
    struct task_struct  *idle;          /* this CPU's idle thread */

    struct cfs_rq       cfs;            /* CFS/EEVDF sub-runqueue */
    struct rt_rq        rt;             /* RT sub-runqueue */
    struct dl_rq        dl;             /* Deadline sub-runqueue */

    /* load tracking for load balancing */
    struct sched_avg    avg_rt;
    struct sched_avg    avg_dl;

    unsigned long       cpu_capacity;   /* this CPU's compute capacity */
} ____cacheline_aligned_in_smp;        /* one struct per cache line */

/* Access the runqueue for a specific CPU */
// struct rq *rq = cpu_rq(cpu);

The nice value is the traditional Unix mechanism for expressing a task's priority among normal (SCHED_NORMAL) tasks. It ranges from -20 (highest priority, "I'm not being nice, I want more CPU") to +19 (lowest priority, "I'm being very nice, give my CPU time to others"). The default is 0.

Here is the key insight that most explanations miss: nice values are not linear — they are exponential. CFS converts a nice value to a weight, and the weight table is geometric with a ratio of approximately 1.25 between adjacent values. This means each step of one nice unit represents about a 10% change in CPU allocation, and 10 nice units represents roughly 10x the CPU time.

The weight table (from kernel/sched/core.c) for nice values -20 to +19:

{ 88761, 71755, 56483, 46273, 36291,
  29154, 23254, 18705, 14949, 11916,
   9548,  7620,  6100,  4904,  3906,
   3121,  2501,  1991,  1586,  1277,
   1024,   820,   655,   526,   423,
    335,   272,   215,   172,   137 }

Nice 0 has weight 1024. If task A has nice 0 (weight 1024) and task B has nice 5 (weight 3906... wait, that's for nice -5), let's do a real example:

• Task A: nice 0, weight = 1024
• Task B: nice 5, weight = 335
• Total weight = 1359
• A's CPU share = 1024/1359 ≈ 75%
• B's CPU share = 335/1359 ≈ 25%

CFS uses weights to compute a task's virtual runtime (vruntime). Instead of incrementing vruntime by actual elapsed nanoseconds, CFS scales it:

delta_vruntime = delta_exec × (NICE_0_WEIGHT / task_weight)

A higher-weight (lower nice, higher priority) task's vruntime grows slower than a lower-weight task's vruntime. Since CFS always picks the task with the smallest vruntime, higher-priority tasks get picked more often. Elegantly, the same data structure and algorithm serves all normal tasks — priority is just a knob that adjusts the rate at which vruntime grows.

/* Demonstrate weight-scaled vruntime calculation (kernel/sched/fair.c style) */
#include <stdio.h>
#include <stdint.h>

/* Weight table: index = nice + 20 */
static const int prio_to_weight[40] = {
 /* -20 */  88761,  71755,  56483,  46273,  36291,
 /* -15 */  29154,  23254,  18705,  14949,  11916,
 /* -10 */   9548,   7620,   6100,   4904,   3906,
 /*  -5 */   3121,   2501,   1991,   1586,   1277,
 /*   0 */   1024,    820,    655,    526,    423,
 /*   5 */    335,    272,    215,    172,    137,
 /*  10 */    110,     87,     70,     56,     45,
 /*  15 */     36,     29,     23,     18,     15,
};

#define NICE_0_WEIGHT 1024

/* How much does vruntime advance for delta_exec nanoseconds? */
uint64_t calc_delta_vruntime(uint64_t delta_exec, int nice) {
    int weight = prio_to_weight[nice + 20];
    /* Scale: higher weight → vruntime grows slower → task runs more */
    return (delta_exec * NICE_0_WEIGHT) / weight;
}

int main(void) {
    uint64_t delta_exec = 1000000; /* 1ms in nanoseconds */
    printf("nice  0 (weight %5d): vruntime += %lu ns\n",
           prio_to_weight[20], calc_delta_vruntime(delta_exec, 0));
    printf("nice +5 (weight %5d): vruntime += %lu ns\n",
           prio_to_weight[25], calc_delta_vruntime(delta_exec, 5));
    printf("nice -5 (weight %5d): vruntime += %lu ns\n",
           prio_to_weight[15], calc_delta_vruntime(delta_exec, -5));
    return 0;
}
/* Output:
 * nice  0 (weight  1024): vruntime += 1000000 ns
 * nice +5 (weight   335): vruntime += 3055223 ns  (grows 3x faster)
 * nice -5 (weight  3121): vruntime +=  328099 ns  (grows 3x slower)
 */

The Completely Fair Scheduler's core idea is the simulation of an "ideal, perfectly fair" CPU. Imagine a magical processor that could run all N tasks simultaneously, each at speed 1/N. Every task would make equal progress at all times. CFS approximates this ideal using a single accounting variable per task: vruntime (virtual runtime), measured in nanoseconds.

The invariant CFS maintains: the task with the smallest vruntime is the one that has been most "cheated" by reality relative to the ideal fair CPU, and it should run next.

CFS stores all runnable tasks in a red-black tree (an approximately balanced binary search tree) keyed by vruntime. The leftmost node of the tree is always the task with the smallest vruntime — and since red-black trees maintain O(log N) balance, finding the next task is O(log N) but in practice the kernel caches a pointer to the leftmost node, making pick_next_task O(1).

When a task runs:

1. Its vruntime is incremented by delta_exec × (NICE_0_WEIGHT / task_weight) (see nice values section).
2. It remains the current task as long as its vruntime stays smaller than the leftmost node in the tree plus a threshold called the scheduling latency (historically called sysctl_sched_latency, typically ~6ms).
3. When it exceeds the threshold, it gets preempted and reinserted into the tree at its new position.

A subtle but important bookkeeping trick: when a task sleeps and wakes up, its vruntime might be far behind the rest of the tree (it wasn't running, so it wasn't accumulating vruntime). If we simply let it run from its old vruntime, it would dominate the CPU for a long time after waking. CFS handles this by updating the sleeping task's vruntime to max(task->vruntime, min_vruntime - threshold) when it wakes, where min_vruntime is a monotonically increasing base that tracks the smallest vruntime in the tree. The task gets a bonus for waking (it runs promptly) but not an unlimited free pass.

The min_vruntime is also used to handle task migration between CPUs: when a task moves to a new CPU, its vruntime is adjusted relative to the new CPU's min_vruntime so it doesn't either steal time or get starved.

/* Conceptual sketch of CFS pick_next_task and task_tick */

/* The CFS runqueue: contains a red-black tree of sched_entities */
struct cfs_rq {
    struct rb_root_cached   tasks_timeline; /* rb-tree; cached leftmost */
    struct sched_entity     *curr;          /* currently running se */
    u64                     min_vruntime;   /* monotonically increasing floor */
    unsigned long           nr_running;
};

/* Each task has a sched_entity embedded in task_struct */
struct sched_entity {
    struct load_weight  load;      /* weight (from nice value) */
    struct rb_node      run_node;  /* our node in the rb-tree */
    u64                 vruntime;  /* virtual runtime in ns */
    u64                 exec_start;/* when we last started running */
    u64                 sum_exec_runtime; /* total actual CPU time */
};

/* Called every timer tick (~4ms): should we preempt? */
static void entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
{
    /* Update vruntime based on time since exec_start */
    update_curr(cfs_rq);  /* curr->vruntime += scaled delta */

    /* Is there a worthier task? */
    if (cfs_rq->nr_running > 1)
        check_preempt_tick(cfs_rq, curr);
}

/* check_preempt_tick: preempt if the leftmost task's vruntime
 * is more than sched_latency behind curr's vruntime */
static void check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
{
    struct sched_entity *se = __pick_first_entity(cfs_rq); /* leftmost */
    s64 delta = (s64)(curr->vruntime - se->vruntime);
    if (delta > sysctl_sched_latency)
        resched_curr(rq_of(cfs_rq));  /* set TIF_NEED_RESCHED */
}

CFS, despite its elegance, has a known flaw: while it is fair on average, it can have high worst-case latency. Consider 100 tasks all with the same vruntime. CFS picks one arbitrarily and lets it run for a full timeslice. The last task to be picked could wait nearly 100 × timeslice before getting any CPU. For latency-sensitive workloads (audio, video, interactive), this is unacceptable.

EEVDF, introduced by Valentin Afanasiev and merged in Linux 6.6 (late 2023), addresses this by adding the concept of a virtual deadline to each task. The algorithm is based on a 1995 paper by Ion Stoica and Hussein Abdel-Wahab.

Each runnable task has:

• vruntime (ve): the virtual time at which the task became eligible to run
• deadline (vd): ve + timeslice/weight — when the task's current "request" is due

A task is eligible if ve <= current_virtual_time. Among all eligible tasks, EEVDF picks the one with the earliest deadline (smallest vd). This is similar to EDF (Earliest Deadline First) scheduling but operating in virtual-time space.

The key insight is that the deadline gives each task a guaranteed latency bound rather than just a long-term fairness guarantee. A task that was recently preempted has a deadline in the near future; it will be picked promptly regardless of how many other tasks are running.

In the kernel, EEVDF still uses a red-black tree (actually now an augmented rb_tree that supports efficient queries of the form "find the eligible task with earliest deadline"). The augmentation stores the minimum deadline in each subtree, allowing the query to be answered in O(log N) time.

For userspace, EEVDF is mostly transparent. The nice value and weights work the same way. The new sched_attr.sched_runtime field (settable via sched_setattr(2)) lets applications hint their preferred timeslice, which directly influences their deadline calculation. A real-time audio application might request a 1ms timeslice, guaranteeing it will be scheduled every few milliseconds even under high load.

/* Conceptual EEVDF pick_next_task logic */

struct sched_entity {
    /* ... existing fields ... */
    u64     deadline;    /* virtual deadline = vruntime + slice/weight */
    u64     slice;       /* requested timeslice (from sched_attr or default) */
};

/*
 * An entity is eligible if its vruntime <= cfs_rq->min_vruntime.
 * Among eligible entities, pick the one with earliest deadline.
 *
 * The rb_tree is augmented: each node stores the minimum deadline
 * in its subtree, so we can skip ineligible subtrees efficiently.
 */
static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
{
    struct rb_node *node = cfs_rq->tasks_timeline.rb_root.rb_node;
    struct sched_entity *se, *best = NULL;

    while (node) {
        se = rb_entry(node, struct sched_entity, run_node);
        /* Is this entity eligible? */
        if (entity_eligible(cfs_rq, se)) {
            /* Is this the earliest deadline so far? */
            if (!best || se->deadline < best->deadline)
                best = se;
            /* Check left subtree (earlier deadlines) */
            node = node->rb_left;
        } else {
            /* Entity not yet eligible; check right subtree */
            node = node->rb_right;
        }
    }
    return best;
}

/* Set the timeslice hint for a task (via sched_setattr syscall) */
/*
 * struct sched_attr attr = {
 *     .size          = sizeof(attr),
 *     .sched_policy  = SCHED_OTHER,   // SCHED_NORMAL
 *     .sched_runtime = 1000000,       // 1ms timeslice hint (ns)
 * };
 * sched_setattr(0, &attr, 0);
 */

A scheduler is only useful if it can actually take the CPU away from a running task. This is preemption. Linux has several preemption modes, and understanding them is key to understanding scheduling latency.

The TIF_NEED_RESCHED flag: The scheduler doesn't immediately preempt a task when it decides to. Instead, it sets a flag TIF_NEED_RESCHED in the current task's thread info. This flag is checked at specific preemption points:

1. Return from a system call — before returning to userspace, the kernel checks the flag.
2. Return from an interrupt handler — same.
3. Explicit schedule() call — when a task voluntarily blocks (e.g., waiting for I/O).
4. Kernel preemption points — if kernel preemption is enabled.

Kernel preemption modes (configured at build time):

• PREEMPT_NONE — the kernel is non-preemptible except at explicit schedule() calls. Lowest latency overhead, but tasks can be stuck in the kernel for a long time. Used in server/throughput builds.
• PREEMPT_VOLUNTARY — adds extra might_resched() checks at common long-running kernel code paths. A compromise.
• PREEMPT (standard) — the kernel can be preempted anywhere that is not holding a spinlock or in an interrupt handler. The preempt_count per-CPU variable tracks nesting.
• PREEMPT_RT (PREEMPT_FULL) — the real-time patch set; almost everything is preemptible, including most interrupt handlers (converted to kernel threads). Achieves sub-millisecond worst-case latency. Now merged into mainline as of Linux 6.12.

The preempt_count mechanism: The kernel tracks preemption disabling through a per-CPU counter. preempt_disable() increments it; preempt_enable() decrements it and checks the reschedule flag. Spinlocks implicitly disable preemption (since spinning while holding a lock and getting preempted would be bad). When preempt_count reaches zero and TIF_NEED_RESCHED is set, __schedule() is called.

This layered mechanism is why the scheduler can safely operate on its own data structures without deadlocking — CFS code runs with preemption disabled and the runqueue spinlock held.

/* Observing scheduling latency from userspace */
#include <stdio.h>
#include <time.h>
#include <sched.h>

/* Measure how long it takes to get the CPU after yielding */
int main(void) {
    struct timespec t1, t2;
    long delta_ns;

    /* Become real-time to reduce noise (requires CAP_SYS_NICE) */
    /* struct sched_param sp = { .sched_priority = 1 };
     * sched_setscheduler(0, SCHED_FIFO, &sp); */

    clock_gettime(CLOCK_MONOTONIC, &t1);
    sched_yield();  /* voluntarily give up CPU; we are immediately re-queued */
    clock_gettime(CLOCK_MONOTONIC, &t2);

    delta_ns = (t2.tv_sec - t1.tv_sec) * 1000000000L
             + (t2.tv_nsec - t1.tv_nsec);
    printf("Scheduling round-trip latency: %ld ns (%ld µs)\n",
           delta_ns, delta_ns / 1000);
    return 0;
}
/* Typical output on a lightly loaded desktop:
 * SCHED_NORMAL: ~50-200µs
 * SCHED_FIFO:   ~10-50µs
 * PREEMPT_RT kernel: ~20-100µs worst-case even under load
 */

Normal tasks share the CPU fairly. Real-time tasks demand priority or timeliness. Linux provides three real-time policies:

SCHED_FIFO (First In, First Out): A FIFO task at priority P runs until it either blocks, exits, or is preempted by a higher-priority task. There is no timeslice — it runs indefinitely if it never blocks. Multiple FIFO tasks at the same priority form a FIFO queue; the one at the head runs until it yields. This is the most aggressive policy: a FIFO task at priority 99 can starve the entire system. That's intentional — if your real-time task doesn't finish, something is genuinely broken.

SCHED_RR (Round Robin): Like FIFO but with a timeslice. When the timeslice expires, the task goes to the back of the queue for its priority level. Other tasks at the same priority get a turn. Tasks at lower priorities still cannot preempt. sched_rr_get_interval(2) returns the timeslice (typically 100ms).

SCHED_DEADLINE: This is the most sophisticated real-time policy, implementing CBS (Constant Bandwidth Server). You specify three parameters:

• sched_runtime (C): the computation time needed per period
• sched_deadline (D): the absolute deadline within each period
• sched_period (T): the period length

The kernel enforces bandwidth: a SCHED_DEADLINE task with C=1ms, T=10ms gets at most 10% of the CPU. If it runs longer than C in a period, it is throttled until the next period. The kernel uses an admission control test (the total bandwidth across all DEADLINE tasks must be ≤ 1) to prevent overcommit.

SCHED_DEADLINE also uses global earliest-deadline-first scheduling across CPUs: a waking deadline task can pull itself to any CPU, preempting any SCHED_NORMAL or SCHED_RT task. This makes deadline scheduling truly global rather than per-CPU.

A safety net: the kernel reserves 5% of CPU time for non-RT tasks by default (/proc/sys/kernel/sched_rt_runtime_us = 950000 out of sched_rt_period_us = 1000000). This prevents a runaway FIFO task from permanently starving the system.

/* Set a thread to SCHED_FIFO priority 50 */
#include <sched.h>
#include <stdio.h>

int set_realtime(int priority) {
    struct sched_param sp = { .sched_priority = priority };
    if (sched_setscheduler(0, SCHED_FIFO, &sp) < 0) {
        perror("sched_setscheduler");
        return -1;
    }
    return 0;
}

/* Set a thread to SCHED_DEADLINE: 1ms work every 10ms */
#include <linux/sched.h>  /* struct sched_attr */
#include <syscall.h>

int set_deadline(void) {
    struct sched_attr attr = {
        .size          = sizeof(attr),
        .sched_policy  = SCHED_DEADLINE,
        .sched_runtime = 1000000,   /* 1ms in nanoseconds */
        .sched_deadline= 5000000,   /* deadline: 5ms after activation */
        .sched_period  = 10000000,  /* period: 10ms */
    };
    if (syscall(SYS_sched_setattr, 0, &attr, 0) < 0) {
        perror("sched_setattr");
        return -1;
    }
    return 0;
}
/* After set_deadline(), if the task runs longer than 1ms in a period,
 * the kernel throttles it. sched_yield() advances to the next period. */

A scheduler that only assigns tasks to fixed CPUs would be useless on modern multi-core hardware — one CPU could be idle while another is saturated with 16 tasks. Load balancing is the mechanism by which Linux redistributes tasks across CPUs to keep them all busy.

Load balancing in Linux is complex because modern CPUs are not a flat set of identical cores. They form a topology tree — cores share L1/L2 cache (NUMA node siblings), NUMA nodes share L3 cache and memory channels, and different sockets share only the interconnect. Moving a task has different costs at each level.

The topology is captured in the scheduling domain (sched_domain) hierarchy. Each CPU belongs to a series of nested domains:

1. SMT domain (sibling hyperthreads on the same core)
2. MC domain (cores on the same physical package)
3. NUMA domain (packages on the same NUMA node)
4. System domain (all CPUs)

Load balancing runs periodically (triggered by the scheduler tick) and checks whether the busiest group in a domain is sufficiently more loaded than the current CPU's group. If so, it calls load_balance() to migrate tasks. The key metric is load — not just task count, but the sum of task weights accounting for the CPU's capacity.

The new load metric: PELT (Per-Entity Load Tracking): Starting with Linux 3.8, load is not just the instantaneous task count. Each sched_entity tracks an exponentially-weighted moving average (EWMA) of its CPU utilization called PELT. This captures the task's historical load contribution, making migration decisions more stable (a task that just woke up doesn't immediately look as heavy as a task that has been running for seconds).

Idle load balancing: When a CPU goes idle, rather than waiting for the next tick, it immediately checks if it can steal tasks from the busiest CPU in its domain. This idle stealing is crucial for latency — it means work gets distributed quickly rather than waiting up to one tick interval.

Task affinity: sched_setaffinity(2) lets applications or administrators pin tasks to specific CPUs. The kernel respects this affinity mask and will never migrate the task outside it. This is useful for NUMA-aware applications and real-time tasks that need deterministic cache behavior.

/* Inspect scheduler topology and task migration statistics */

/* 1. View CPU topology */
// cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list
// cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list

/* 2. View per-CPU runqueue statistics */
// cat /proc/schedstat
/* Fields include: number of tasks, yield calls, legacy fields,
 * schedule() calls, idle calls, ttwu (try_to_wake_up) calls */

/* 3. Pin a process to CPU 0 and 1 only */
#include <sched.h>
#include <stdio.h>

int pin_to_cpus(pid_t pid) {
    cpu_set_t mask;
    CPU_ZERO(&mask);
    CPU_SET(0, &mask);
    CPU_SET(1, &mask);
    if (sched_setaffinity(pid, sizeof(mask), &mask) < 0) {
        perror("sched_setaffinity");
        return -1;
    }
    printf("Process %d pinned to CPUs 0-1\n", pid);
    return 0;
}

/* 4. Use perf to observe scheduler events */
// sudo perf sched record -- sleep 10
// sudo perf sched latency
// sudo perf sched timehist
/* timehist shows per-task scheduling latency and sleep/run durations */

Consider a server running containers. Container A has 8 web server processes, container B has 1 database process. With a flat CFS, A gets 8/9 of the CPU and B gets 1/9 — unfair to B's owner, who may be paying for equal CPU share.

CFS task groups solve this by introducing a two-level hierarchy. Rather than scheduling individual tasks, the top level schedules groups of tasks. Each group gets a CPU share proportional to its weight, and within each group, its tasks are scheduled fairly among themselves.

This is implemented via the cgroup (control group) v2 interface. The cpu cgroup controller provides:

• cpu.weight — the relative weight of this cgroup (analogous to nice value for groups, range 1–10000, default 100)
• cpu.max — absolute bandwidth limit: quota period e.g. 500000 1000000 means the cgroup can use at most 50% of one CPU per second (CFS bandwidth control)
• cpu.stat — scheduler statistics for the cgroup

The implementation uses group scheduling entities — a sched_entity that represents the entire group in the parent CFS runqueue. When the group entity is scheduled, it internally runs its own CFS runqueue to pick a task within the group.

This creates a tree of CFS runqueues: the root CFS runqueue has one entry per top-level cgroup (plus uncgrouped tasks); each cgroup's CFS runqueue has entries for its sub-cgroups and tasks. The scheduler does a two-pass pick: pick the group, then pick a task within the group.

For containers and systemd services, this is managed automatically. You can inspect it via:

cat /sys/fs/cgroup/system.slice/cpu.stat

# Demonstrate cgroup CPU control (run as root or with appropriate caps)

# Create two cgroups with different weights
mkdir -p /sys/fs/cgroup/demo_high /sys/fs/cgroup/demo_low

# High priority: weight 400 (4x the default of 100)
echo 400 > /sys/fs/cgroup/demo_high/cpu.weight

# Low priority: weight 100 (default)
echo 100 > /sys/fs/cgroup/demo_low/cpu.weight

# Apply a bandwidth limit to the low-priority group:
# max 200ms of CPU per 1000ms period (20% of one core)
echo "200000 1000000" > /sys/fs/cgroup/demo_low/cpu.max

# Move processes into the cgroups
echo $$ > /sys/fs/cgroup/demo_high/cgroup.procs  # current shell

# Run a CPU-heavy task in the low-priority cgroup
# echo <pid> > /sys/fs/cgroup/demo_low/cgroup.procs

# Observe CPU allocation
# watch -n 1 'cat /sys/fs/cgroup/demo_high/cpu.stat; cat /sys/fs/cgroup/demo_low/cpu.stat'

You cannot improve what you cannot measure. Linux exposes rich scheduler observability through multiple interfaces.

/proc/sched_debug: A comprehensive dump of scheduler state including per-CPU runqueues, task vruntimes, EEVDF deadlines, load statistics, and tunable parameters. The sched_debug=1 kernel boot parameter enables even more detail. This is the first place to look when debugging a scheduler anomaly.

/proc/<pid>/sched: Per-task scheduling statistics — number of context switches, total run time, sleep time, wait time (time runnable but not running, i.e., scheduling latency), number of times migrated between CPUs.

perf sched: The perf tool can record all scheduler events (context switches, wakeups, migrations) via kernel tracepoints and replay or analyze them. perf sched latency gives a histogram of scheduling latencies per task; perf sched timehist gives a timeline.

ftrace: The kernel's built-in tracing infrastructure. The sched_switch, sched_wakeup, and sched_migrate_task tracepoints can be enabled via /sys/kernel/tracing/events/sched/. trace-cmd and KernelShark provide higher-level interfaces.

Tunable sysctls (/proc/sys/kernel/sched_*):

• sched_latency_ns (default ~6ms): the target scheduling latency — the time by which every runnable task should get at least one turn. Decreasing this improves latency at the cost of more context switches.
• sched_min_granularity_ns (default ~750µs): minimum time a task runs before it can be preempted. Prevents thrashing.
• sched_migration_cost_ns (default ~500µs): the estimated cost of migrating a task between CPUs. If a task ran within this window, it is considered "cache hot" and the load balancer avoids migrating it.
• sched_nr_migrate (default 32): maximum tasks migrated per load-balance call.

For the EEVDF scheduler, the sched_latency_ns tunable takes on a slightly different meaning (it bounds the worst-case deadline), but the intent is the same.

# Practical scheduler observability commands

# 1. Current scheduler state (per-CPU runqueues, vruntimes)
sudo cat /proc/sched_debug | head -80

# 2. Per-task scheduling stats for a process
cat /proc/$(pgrep firefox)/sched
# Key fields: nr_switches (context switches), wait_sum (total time runnable but waiting)

# 3. Record and analyze scheduling latency with perf
sudo perf sched record -a -- sleep 10   # record all CPUs for 10s
sudo perf sched latency                  # show per-task latency histogram
sudo perf sched timehist | head -50      # timeline of context switches

# 4. Watch scheduler tunables
sysctl kernel.sched_latency_ns
sysctl kernel.sched_min_granularity_ns
sysctl kernel.sched_migration_cost_ns

# 5. Enable the sched_switch tracepoint via ftrace
cd /sys/kernel/tracing
echo 1 > events/sched/sched_switch/enable
echo 1 > tracing_on
sleep 2
echo 0 > tracing_on
cat trace | head -40
# Shows: prev_comm, prev_pid, prev_prio, next_comm, next_pid