Real-Time Linux (PREEMPT_RT)

Before writing a line of kernel code, you must internalize the distinction between throughput and latency, and understand why optimizing one often harms the other.

Throughput is how much work the system finishes per unit time. A web server that processes 100,000 requests per second has high throughput. To maximize throughput, you batch work, amortize overhead, and keep CPUs busy — even if some individual requests occasionally wait longer.

Latency is how long a specific piece of work takes from trigger to completion. A real-time audio system that must fill a 128-sample buffer at 48 kHz has a deadline of 2.67 ms. Missing it means an audible glitch. Latency here is the only metric that matters.

Worst-case latency (WCL) is what real-time engineers care about, not average latency. A system that handles 99.99% of events in 1 µs but occasionally takes 50 ms is useless for hard real-time. This is the key insight: real-time is about predictability and bounded worst-case behavior, not speed.

The categories of real-time requirements:

• Hard real-time: Missing a deadline causes system failure (airbag controller, pacemaker). Zero tolerance.
• Firm real-time: Missing a deadline makes the result useless but does not cause failure (video frame decoder — drop the frame).
• Soft real-time: Deadlines are targets; degraded quality results from misses (audio playback, interactive UI).

A standard Linux kernel is optimized for throughput and average-case latency. Its design decisions — long non-preemptible critical sections, batched interrupt processing, work deferral via softirqs — are excellent for this goal but create unpredictable latency spikes for real-time tasks.

/* Measure scheduling latency from userspace using clock_nanosleep */
#include <stdio.h>
#include <time.h>
#include <stdint.h>

#define NSEC_PER_SEC 1000000000LL
#define ITERATIONS   100000

int main(void) {
    struct timespec next, now;
    long max_lat = 0, lat;

    clock_gettime(CLOCK_MONOTONIC, &next);

    for (int i = 0; i < ITERATIONS; i++) {
        /* Advance target by 1 ms */
        next.tv_nsec += 1000000;
        if (next.tv_nsec >= NSEC_PER_SEC) {
            next.tv_nsec -= NSEC_PER_SEC;
            next.tv_sec++;
        }

        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
        clock_gettime(CLOCK_MONOTONIC, &now);

        /* How late are we? */
        lat = (now.tv_sec - next.tv_sec) * NSEC_PER_SEC
            + (now.tv_nsec - next.tv_nsec);
        if (lat > max_lat) max_lat = lat;
    }

    printf("Max latency over %d iterations: %ld ns\n",
           ITERATIONS, max_lat);
    return 0;
}
/* Compile: gcc -O2 -o latency latency.c
 * On a stock kernel you may see max_lat > 1,000,000 ns (1 ms)
 * On PREEMPT_RT with CPU isolation: often < 50,000 ns */

Linux supports multiple preemption models, selectable at build time via CONFIG_PREEMPTION. Understanding each model's semantics is essential before you can understand what PREEMPT_RT changes.

No preemption (CONFIG_PREEMPT_NONE): The kernel is non-preemptible. A process running in kernel mode (e.g., in the middle of a system call) can only be preempted at explicit yield points — when it blocks, calls schedule(), or returns to user space. This maximizes throughput and is the default for server kernels. Worst-case latency can be hundreds of milliseconds.

Voluntary preemption (CONFIG_PREEMPT_VOLUNTARY): Adds explicit might_sleep() and cond_resched() calls at strategic points in the kernel to create voluntary preemption points. Improves latency somewhat at minimal throughput cost. Still not suitable for real-time.

Full preemption (CONFIG_PREEMPT): The kernel is preemptible at almost any point outside of spinlock-protected regions and interrupt handlers. A high-priority task can preempt a lower-priority task in the middle of a system call, as long as the kernel is not holding a spinlock. Suitable for desktop/embedded interactive use. Worst-case latency drops to single-digit milliseconds.

PREEMPT_RT (CONFIG_PREEMPT_RT): The kernel is preemptible at essentially every point, including inside what were previously non-preemptible spinlock regions and interrupt handlers. This is achieved by converting spinlocks to sleeping mutexes and making interrupt handlers run as kernel threads. Worst-case latency drops to tens of microseconds on well-tuned hardware.

The critical invariant the kernel maintains is the preempt count. Every CPU has a per-CPU preempt_count variable. When it is non-zero, preemption is disabled. spin_lock() increments it; spin_unlock() decrements it and may trigger a reschedule. PREEMPT_RT's key move is to replace this increment/decrement with a mutex lock/unlock — so the 'disabled preemption' period becomes 'holding a mutex', which allows other tasks to sleep-wait.

/* Check preemption depth from inside the kernel */
#include <linux/preempt.h>

void example_kernel_function(void)
{
    /* preempt_count() returns the current nesting depth:
     * 0           = fully preemptible
     * bits 0-7    = preempt_disable() nesting count
     * bits 8-15   = softirq nesting
     * bits 16-27  = hardirq nesting
     * bit 28      = NMI context
     */
    if (in_atomic())
        pr_info("Non-preemptible context, depth=%d\n",
                preempt_count());

    /* On PREEMPT_RT, spin_lock() no longer increments
     * preempt_count for the 'lock held' bits — it sleeps.
     * Only raw_spin_lock() still increments it.
     */
    preempt_disable();   /* Explicitly disable preemption */
    /* ... small critical section ... */
    preempt_enable();    /* Re-enable; may call schedule() */
}

The PREEMPT_RT patchset (developed primarily by Ingo Molnar, Thomas Gleixner, and Steven Rostedt, maintained for years out-of-tree before mainline merger) makes a precise set of changes. Understanding each change and why it helps is the core of this module.

1. Sleeping spinlocks (RT mutexes): In a standard kernel, spin_lock() busy-waits (spins) in a tight loop until the lock is free. During this time, preemption is disabled. If a high-priority task needs the CPU, it cannot get it — the spinning task holds the CPU hostage.

PREEMPT_RT replaces spinlock_t with an rt_mutex (a priority-inheritance mutex) under the hood. Instead of spinning, a task that cannot acquire the lock sleeps and is removed from the run queue. The scheduler is now free to run other, higher-priority tasks. When the lock is released, the sleeping task is woken up.

The caveat: some spinlocks cannot be made sleeping. Locks held in NMI handlers, or in the core scheduler itself, must remain true spinlocks. These are called raw_spinlock_t and are the explicit escape hatch that drivers use when they genuinely cannot sleep. On PREEMPT_RT, using raw_spinlock_t where a sleeping lock would have been acceptable is a bug.

2. Threaded interrupt handlers: In a standard kernel, hardware interrupts run in a special non-preemptible top-half context, completely outside of the normal scheduler. A high-priority SCHED_FIFO task is powerless against a flood of network interrupts — the interrupt handler simply takes the CPU.

PREEMPT_RT converts nearly all interrupt handlers into kernel threads (using kthread infrastructure). Each IRQ gets a irq/N-name kernel thread. These threads participate in normal priority-based scheduling. A real-time task at priority 90 will preempt an IRQ thread at the default priority 50. You can use chrt and taskset to tune IRQ thread priorities.

3. High-resolution timers (hrtimers): The standard jiffies-based timer wheel has resolution limited to the HZ rate (typically 250 Hz = 4 ms). PREEMPT_RT requires CONFIG_HIGH_RES_TIMERS, which programs hardware one-shot timers (APIC timer on x86, generic timer on ARM) for nanosecond-resolution wakeups. clock_nanosleep(TIMER_ABSTIME) becomes genuinely precise.

4. PI-futexes (Priority-Inheritance Futexes): User-space real-time programs use POSIX mutexes. PREEMPT_RT extends priority inheritance to the futex system call, so user-space priority inversion is also solved. This requires PTHREAD_PRIO_INHERIT protocol in the mutex attributes.

/* Check at runtime whether the kernel has PREEMPT_RT */
#include <stdio.h>
#include <string.h>

int is_preempt_rt(void)
{
    /* PREEMPT_RT kernels include "PREEMPT_RT" in /proc/version
     * or in uname -v output */
    FILE *f = fopen("/proc/version", "r");
    if (!f) return -1;
    char buf[256];
    fgets(buf, sizeof(buf), f);
    fclose(f);
    return strstr(buf, "PREEMPT_RT") != NULL;
}

/* Shell equivalent:
 * uname -v | grep -q PREEMPT_RT && echo "RT kernel" || echo "Standard"
 *
 * Or check kernel config:
 * zcat /proc/config.gz | grep CONFIG_PREEMPT_RT
 */

int main(void)
{
    int rt = is_preempt_rt();
    printf("PREEMPT_RT: %s\n",
           rt == 1 ? "yes" : rt == 0 ? "no" : "unknown");
    return 0;
}

Priority inversion is the most infamous real-time bug, immortalized by the Mars Pathfinder mission in 1997, where it caused daily system resets on Mars. Understanding it deeply is non-negotiable for real-time engineering.

The scenario: Imagine three tasks:

• Task H (high priority): needs a shared resource protected by mutex M
• Task M (medium priority): CPU-intensive, does not use mutex M
• Task L (low priority): holds mutex M, sleeping inside a critical section

1. Task L acquires mutex M and enters its critical section.
2. Task L is preempted by Task H (which just woke up).
3. Task H tries to acquire mutex M — it is held by Task L, so Task H blocks.
4. Now Task M runs, because it has higher priority than Task L.
5. Task M runs for a long time, blocking Task L from running and releasing the mutex.
6. Task H is effectively blocked by Task M, even though Task H has higher priority than Task M. This is priority inversion.

In the worst case, this inversion is unbounded: any number of medium-priority tasks can delay Task H indefinitely.

Priority inheritance is the solution: when Task H blocks on mutex M (held by Task L), the kernel temporarily boosts Task L's priority to match Task H's. Now Task L runs at Task H's priority, quickly finishes its critical section, releases M, and Task H can proceed. Task L's priority is restored when it releases the mutex.

Linux implements priority inheritance via rt_mutex (which PREEMPT_RT uses for all spinlocks) and via PTHREAD_PRIO_INHERIT mutexes in userspace. The inheritance can be transitive: if Task L is itself blocked on another mutex held by Task X, Task X also gets its priority boosted. This chain can be followed up to MAX_PI_CHAIN_WALK depth (currently 15).

Priority ceiling is an alternative: the mutex has a pre-assigned ceiling priority; any task that acquires it immediately runs at that priority. Simpler to analyze, harder to set correctly. POSIX provides PTHREAD_PRIO_PROTECT for this.

/* Correct POSIX real-time mutex with priority inheritance */
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    pthread_mutex_t mutex;
    pthread_mutexattr_t attr;

    pthread_mutexattr_init(&attr);

    /* Request priority inheritance protocol.
     * Without this, on a real-time system, priority inversion
     * is possible. PREEMPT_RT kernels support this via PI-futex. */
    pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_INHERIT);

    /* Also useful: PTHREAD_PRIO_PROTECT (priority ceiling) */
    /* pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_PROTECT);
     * pthread_mutexattr_setprioceiling(&attr, 80); */

    if (pthread_mutex_init(&mutex, &attr) != 0) {
        perror("mutex_init — needs CAP_SYS_NICE or PREEMPT_RT");
        return 1;
    }

    pthread_mutexattr_destroy(&attr);
    /* Use mutex normally... */
    pthread_mutex_lock(&mutex);
    /* critical section */
    pthread_mutex_unlock(&mutex);

    pthread_mutex_destroy(&mutex);
    return 0;
}
/* Compile: gcc -O2 -pthread -o pi_mutex pi_mutex.c */

Understanding threaded IRQs requires first understanding how interrupt handling works in a standard kernel.

Standard interrupt handling has two halves:

• Top half (hardirq context): Runs immediately when hardware asserts the interrupt line. Preemption is disabled. IRQs are disabled on the local CPU. It must finish in microseconds. It typically acknowledges the hardware and schedules the bottom half.
• Bottom half (softirq / tasklet / workqueue): Runs after the top half, with more flexibility. Softirqs still run with preemption disabled; workqueues run in normal process context.

The problem: the top half is completely outside the scheduler's control. No task, regardless of priority, can preempt it.

With PREEMPT_RT, the IRQF_NO_THREAD flag is the exception; most interrupt handlers run as kernel threads. The kernel creates a irq/N-drivername kthread for each IRQ. When hardware fires:

1. A minimal hardirq handler (the true top half) runs, masked from further interrupts, does the bare minimum (ack the hardware), and wakes the IRQ thread.
2. The IRQ thread runs as a normal schedulable entity. It can be preempted by higher-priority threads.

This is why you can do ps aux | grep irq/ on a PREEMPT_RT system and see dozens of kernel threads — one per IRQ line. You can then use chrt to adjust their scheduling policy and priority:

# Find the PID of the ethernet IRQ thread
pgrep -a irq | grep eth0
# Move it to SCHED_FIFO priority 70
chrt -f -p 70 <pid>

The request_threaded_irq() kernel API is what drivers use to explicitly register a threaded handler. The first function argument is the top-half (hardirq-safe) handler; the second is the thread function:

request_threaded_irq(irq_num, my_hardirq_handler,
                     my_thread_handler,
                     IRQF_SHARED, "mydriver", dev);

On a non-RT kernel you can also use threaded IRQs for drivers that do slow work in their interrupt handler (e.g., I2C reads) — this is a clean design that happens to also be PREEMPT_RT-compatible.

/* Minimal kernel module demonstrating threaded IRQ registration.
 * DO NOT run on production hardware — uses a dummy IRQ for illustration. */
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/kernel.h>

/* Top half: runs in hardirq context, must be atomic.
 * On PREEMPT_RT this still runs briefly non-preemptible.
 * Return IRQ_WAKE_THREAD to schedule the thread handler. */
static irqreturn_t my_hardirq_handler(int irq, void *dev_id)
{
    /* Acknowledge hardware, check if this is our interrupt */
    return IRQ_WAKE_THREAD; /* schedule my_thread_handler */
}

/* Thread handler: runs in normal process context (kthread).
 * Can sleep, use mutexes, do I/O. Fully schedulable.
 * On PREEMPT_RT: preemptible by higher-priority RT tasks. */
static irqreturn_t my_thread_handler(int irq, void *dev_id)
{
    /* Safe to sleep, allocate memory, etc. */
    pr_info("Thread handler running, can_sleep=%d\n",
            !in_atomic());
    return IRQ_HANDLED;
}

static int __init my_init(void)
{
    /* In a real driver, irq_num comes from platform data */
    int irq_num = 10; /* placeholder */
    return request_threaded_irq(irq_num,
        my_hardirq_handler,
        my_thread_handler,
        IRQF_SHARED,
        "mydriver",
        (void *)my_init /* dev_id */
    );
}
MODULE_LICENSE("GPL");

Linux implements the POSIX real-time scheduling policies on top of its general-purpose CFS (Completely Fair Scheduler). RT tasks are handled by a separate scheduling class (rt_sched_class) that has strict priority over CFS tasks.

SCHED_FIFO (First In, First Out): A SCHED_FIFO task at priority P runs until:

• It voluntarily sleeps or blocks on I/O
• It calls sched_yield()
• A task with higher priority P' > P becomes runnable

It will never be preempted by a task at the same or lower priority. This is the policy you use for hard real-time work: your task either runs or sleeps, and no other RT task of equal priority steals your CPU.

SCHED_RR (Round Robin): Like SCHED_FIFO, but adds a time quantum. If a SCHED_RR task exhausts its quantum without sleeping, it is placed at the back of the queue for its priority level, and the next SCHED_RR task at the same priority runs. Useful for multiple RT tasks that should share the CPU fairly.

Priority levels: RT priorities are integers 1–99 (kernel internal: 99–1, inverted to match POSIX). Priority 99 is the highest. The kernel reserves priorities 98–99 for watchdog and migration threads. User-space RT tasks typically use priorities 1–90.

The RT throttle: A crucial safety net. By default, RT tasks cannot consume more than 95% of CPU time (configurable via /proc/sys/kernel/sched_rt_runtime_us and sched_rt_period_us). If they do, the kernel backs off and lets CFS tasks run. On a fully configured RT system, you can set sched_rt_runtime_us = -1 to disable this — but then a runaway RT task can lock up the system. Know what you are doing before disabling it.

SCHED_DEADLINE: A newer policy (EDF-based) where you specify runtime, deadline, and period. The kernel guarantees your task gets runtime CPU time every period, with the deadline constraint. More expressive than SCHED_FIFO but harder to combine with PREEMPT_RT — it is mostly for soft real-time video/audio work.

/* Set a thread to SCHED_FIFO priority 80 */
#include <pthread.h>
#include <sched.h>
#include <stdio.h>
#include <errno.h>

void set_realtime(int priority)
{
    struct sched_param param;
    param.sched_priority = priority;

    /* Set current thread to SCHED_FIFO.
     * Requires CAP_SYS_NICE or RLIMIT_RTPRIO set appropriately.
     * On modern systems: set via /etc/security/limits.conf
     *   @realtime  -  rtprio  99
     * Or run as root. */
    if (sched_setscheduler(0, SCHED_FIFO, &param) == -1) {
        perror("sched_setscheduler");
        return;
    }
    printf("Now running SCHED_FIFO priority %d\n", priority);
}

/* Lock memory to prevent page faults during RT operation */
void lock_memory(void)
{
    #include <sys/mman.h>
    if (mlockall(MCL_CURRENT | MCL_FUTURE) == -1)
        perror("mlockall");  /* page faults are latency killers */
}

/* Shell:
 * chrt -f 80 ./my_rt_program       # launch with SCHED_FIFO 80
 * chrt -p <pid>                     # query policy/priority
 * cat /proc/sys/kernel/sched_rt_runtime_us  # RT throttle (950000 = 95%) */

Even with PREEMPT_RT enabled, a real-time task competing with unrelated kernel work on the same CPU will see latency spikes. The solution is CPU isolation: dedicate one or more CPUs exclusively to real-time tasks, and move everything else off them.

Kernel boot parameter isolcpus=: The most powerful isolation tool. Adding isolcpus=2,3 to the kernel command line removes CPUs 2 and 3 from the general scheduler's pool. The kernel's load balancer will not move tasks onto those CPUs, and IRQs will not be routed there by default. Your RT task, pinned to CPU 2 with taskset or pthread_setaffinity_np(), gets the CPU essentially to itself.

nohz_full= (tickless CPUs): By default, the kernel fires a scheduling timer tick (usually at 250 Hz = every 4 ms) on every CPU. This tick itself is an interrupt that can perturb an RT task. nohz_full=2,3 disables the tick on CPUs 2 and 3 when only one runnable task is present. Combined with isolcpus, this is the gold standard setup.

IRQ affinity: Each IRQ has a smp_affinity bitmask (in /proc/irq/N/smp_affinity) that controls which CPUs can handle it. Move all IRQs off your isolated CPUs:

for i in /proc/irq/*/smp_affinity; do
    echo 1 > $i  # pin all IRQs to CPU 0 (bitmask 0x1)
done

rcu_nocbs=: RCU (Read-Copy-Update) callbacks can fire on any CPU and create latency. rcu_nocbs=2,3 moves RCU callback processing off those CPUs.

The full production incantation looks like:

isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

After booting, verify with cat /sys/devices/system/cpu/isolated and use cyclictest (from the rt-tests package) to measure the actual worst-case scheduling latency you achieve.

#!/bin/bash
# Setup script for RT CPU isolation (run as root)
# Assumes CPUs 2-3 are isolated via kernel command line:
# isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

RT_CPUS="2,3"      # the isolated CPUs
HOUSEKEEPING="0,1" # the non-isolated CPUs

# Move all IRQs to housekeeping CPUs (bitmask: 0x3 = CPUs 0 and 1)
for irq_dir in /proc/irq/*/; do
    echo 3 > "${irq_dir}smp_affinity" 2>/dev/null
done

# Move kernel threads to housekeeping CPUs
for pid in $(ps -eLo pid,comm | grep -E '^\s*[0-9]' | awk '{print $1}'); do
    taskset -cp $HOUSEKEEPING $pid 2>/dev/null
done

# Measure latency with cyclictest on an isolated CPU
# -m: mlockall, -t1: 1 thread, -p99: SCHED_FIFO 99
# -i: 200µs interval, -l: 100000 iterations, -a2: CPU 2
cyclictest -m -t1 -p99 -i 200 -l 100000 -a 2 --histogram=100

# Plot histogram output with gnuplot for a latency distribution curve

You cannot tune what you cannot measure. The real-time Linux ecosystem has mature, precise tooling for measuring scheduling latency.

cyclictest (from the rt-tests package, written by Thomas Gleixner) is the canonical real-time latency measurement tool. It creates a high-priority thread that sleeps for a fixed interval, wakes up, measures how late it woke, and records statistics. Run it with:

cyclictest -m -p99 -t -i200 --duration=60s

The output gives min/avg/max latency per CPU. On a well-tuned PREEMPT_RT system, max latency under realistic load should be under 100 µs. On a stock kernel, max latency under load can exceed 5 ms.

hackbench generates scheduler stress (hundreds of threads communicating via pipes/sockets) and is used while cyclictest runs to create realistic worst-case conditions. Run both simultaneously:

hackbench -l 100 &
cyclictest -m -p99 -t -i200 --duration=60s

ftrace latency tracers: The kernel's built-in ftrace framework includes two tracers specifically for real-time analysis:

• preemptoff: Traces periods when preemption was disabled, showing the worst-case lock-hold times.
• irqsoff: Traces periods when IRQs were disabled, identifying interrupt latency sources.

Usage:

echo preemptoff > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
# ... run workload ...
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace

The trace output shows exactly which function held preemption disabled the longest, with a full call stack. This is how kernel developers find the latency offenders to fix.

osnoise: A newer tracer in rt-tests that measures OS noise (any unwanted interruption of a CPU-bound RT task) directly, giving you a fine-grained latency budget breakdown.

#!/bin/bash
# Automated latency measurement and reporting
# Requires: rt-tests package (cyclictest), stress-ng

set -e
RESULTS="/tmp/rt_latency_$(date +%Y%m%d_%H%M%S).txt"

echo "=== Kernel version ==" | tee $RESULTS
uname -rv | tee -a $RESULTS

echo "=== PREEMPT model ==" | tee -a $RESULTS
zcat /proc/config.gz 2>/dev/null | grep -E 'PREEMPT' | tee -a $RESULTS

# Generate realistic background load
stress-ng --cpu 2 --io 2 --vm 1 --fork 4 &
LOAD_PID=$!

echo "=== cyclictest (60s, SCHED_FIFO 99) ==" | tee -a $RESULTS
# -m: mlockall, -t: one thread/CPU, -p99: SCHED_FIFO 99
# -i200: 200µs intervals, --duration=60s
cyclictest -m -t -p99 -i200 --duration=60s \
    --histogram=200 --histfile=/tmp/hist.dat \
    2>&1 | tee -a $RESULTS

kill $LOAD_PID
wait $LOAD_PID 2>/dev/null

echo "Results saved to $RESULTS"
echo "Histogram data in /tmp/hist.dat (plot with gnuplot)"

When the kernel converted spinlock_t to a sleeping lock under PREEMPT_RT, some code could not be converted. These are the places where sleeping is genuinely impossible — not just inconvenient. The raw_spinlock_t type is the explicit marker for this code.

When must you use raw_spinlock_t?

1. Code that runs in NMI (Non-Maskable Interrupt) context. NMIs cannot sleep; there is no way to reschedule from an NMI handler.
2. Code that implements the scheduler itself. You cannot sleep inside the code that decides who gets to sleep.
3. Code that implements the sleeping lock infrastructure (rt_mutex internals). These locks protect the lock metadata — they cannot themselves be sleeping locks without infinite recursion.
4. Code in early boot, before the scheduler is initialized.

Driver and subsystem code should NOT use raw_spinlock_t unless they have a genuine, documented reason. Using it is a statement: "I cannot sleep here, and I accept that holding this lock will prevent PREEMPT_RT from achieving low latency." PREEMPT_RT reviewers will push back hard on any raw_spinlock_t introduced without justification.

The conversion rules for kernel developers porting code to be PREEMPT_RT compatible:

• spin_lock() → can stay as is; PREEMPT_RT makes it sleeping automatically.
• If you used spin_lock() but are in a context that cannot sleep (IRQ handler, atomic context), either: convert to threaded IRQ (preferred) or switch to raw_spin_lock() (last resort).
• Anything that was spin_lock_irqsave() in non-PREEMPT_RT code needs careful analysis: on PREEMPT_RT, the lock becomes a mutex; the irqsave part still disables local IRQs, which may now be unnecessary overhead.

The practical implication for contribution: any patch that introduces a new raw_spinlock_t will require a comment explaining why, and will get extra scrutiny in code review.

/* Illustrating spinlock_t vs raw_spinlock_t semantics */
#include <linux/spinlock.h>
#include <linux/mutex.h>

/* On PREEMPT_RT:
 * spinlock_t    → rt_mutex (can sleep, has priority inheritance)
 * raw_spinlock_t → true spinlock (busy-wait, no sleep, no PI)
 */

static spinlock_t     normal_lock;  /* OK to use everywhere */
static raw_spinlock_t raw_lock;     /* Only for truly-atomic contexts */

void normal_driver_operation(void)
{
    unsigned long flags;

    /* On PREEMPT_RT this will sleep if contended.
     * Safe in process context, threaded IRQ handlers.
     * Do NOT use in NMI, scheduler internals. */
    spin_lock_irqsave(&normal_lock, flags);
    /* ... your critical section ... */
    spin_unlock_irqrestore(&normal_lock, flags);
}

void scheduler_internal_operation(void)
{
    unsigned long flags;

    /* Must be a true spinlock — we ARE the scheduler.
     * This section will briefly make the CPU non-preemptible.
     * Keep this as short as humanly possible. */
    raw_spin_lock_irqsave(&raw_lock, flags);
    /* ... absolutely minimal work ... */
    raw_spin_unlock_irqrestore(&raw_lock, flags);
}

/* PREEMPT_RT contribution rule:
 * If you add raw_spinlock_t, you MUST add a comment:
 * /* Must be a raw spinlock because: [specific reason] */

A page fault is a latency disaster for real-time tasks. When a process accesses memory that is paged out to disk or not yet mapped into physical memory, the CPU takes a page fault exception. The kernel must:

1. Find the missing page (may involve disk I/O — milliseconds)
2. Allocate a physical page frame
3. Load the page contents
4. Update the page tables
5. Return to the faulting instruction

For a real-time task with a 500 µs deadline, a page fault that takes 5 ms is catastrophic. The solution is memory locking: pin all pages in physical RAM before the real-time phase begins.

mlockall(MCL_CURRENT | MCL_FUTURE) locks all current and future pages of the process into RAM. MCL_CURRENT locks everything already mapped; MCL_FUTURE prevents any future allocations from being paged out. This must be called before entering the RT loop.

Heap prefaulting: mlockall prevents pages from being evicted but does not cause unmapped pages (e.g., future malloc regions) to be faulted in immediately. The trick is to explicitly touch all memory you will use, forcing the faults before the RT phase:

/* Prefault the stack */
unsigned char stack_prefault[MAX_STACK_SIZE];
memset(stack_prefault, 0, sizeof(stack_prefault));

/* Prefault the heap */
void *heap = malloc(HEAP_SIZE);
memset(heap, 0, HEAP_SIZE);

Huge pages: For applications with large memory footprints, transparent huge pages (THP) can cause latency spikes when the kernel splits or collapses huge pages in the background. On RT systems, it is often better to disable THP (echo never > /sys/kernel/mm/transparent_hugepage/enabled) and use explicit huge pages if needed.

NUMA considerations: On multi-socket systems, accessing memory on a remote NUMA node has significantly higher latency than local memory. Pin RT tasks to a CPU set and use numactl to ensure memory is allocated locally.

/* Complete RT task initialization: lock memory, set policy, prefault */
#include <sys/mman.h>
#include <pthread.h>
#include <sched.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

#define HEAP_SIZE    (64 * 1024 * 1024)  /* 64 MB */
#define STACK_SIZE   (8  * 1024 * 1024)  /* 8 MB */

void rt_init(int sched_priority)
{
    struct sched_param param = { .sched_priority = sched_priority };

    /* 1. Lock all pages in RAM — call before mlockall. */
    if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0) {
        perror("mlockall");
        exit(1);
    }

    /* 2. Set real-time scheduling policy. */
    if (sched_setscheduler(0, SCHED_FIFO, &param) != 0) {
        perror("sched_setscheduler");
        exit(1);
    }

    /* 3. Prefault the stack (touch every page). */
    {
        unsigned char stack_buf[STACK_SIZE];
        memset(stack_buf, 0, sizeof(stack_buf));
    }

    /* 4. Prefault the heap. */
    {
        void *heap = malloc(HEAP_SIZE);
        if (!heap) { perror("malloc"); exit(1); }
        memset(heap, 0, HEAP_SIZE);
        free(heap);
    }

    printf("RT init complete: SCHED_FIFO prio %d, memory locked\n",
           sched_priority);
}
/* After rt_init(), enter your real-time loop.
 * Avoid: dynamic memory allocation, file I/O,
 *        blocking syscalls (use non-blocking or pre-loaded data),
 *        printf (use a lock-free ring buffer for logging). */

Achieving deterministic scheduling on Linux requires applying all the concepts in this module in concert. Think of it as layers of uncertainty to eliminate, one by one.

Layer 1: Hardware uncertainty

• SMI (System Management Interrupt): x86 CPUs can receive SMIs from the BIOS/UEFI firmware that take the CPU offline for hundreds of microseconds, completely transparent to the OS. Check with rdmsr -a 0x34 (MSR_SMI_COUNT). Reduce with BIOS settings (disable USB legacy emulation, ACPI idle, etc.). Some BIOSes have an RT mode.
• Cache effects: Cold caches cause extra latency. Warm them up in initialization.
• CPU frequency scaling: intel_pstate scaling can cause latency. Fix frequency with cpupower frequency-set -g performance or use the performance governor.

Layer 2: Kernel uncertainty

• Preemption: Solved by PREEMPT_RT.
• Interrupt storms: Solved by IRQ affinity (move IRQs off RT CPUs).
• Timer tick: Solved by nohz_full.
• RCU callbacks: Solved by rcu_nocbs.
• Memory reclaim (kswapd): Can steal CPU. Solved by mlockall + sufficient RAM.

Layer 3: Application uncertainty

• Dynamic memory allocation: Use a pool allocator in the RT loop.
• I/O: Load all data before the RT phase; use memory-mapped files.
• Logging: Use a lock-free ring buffer; drain it from a non-RT thread.
• System calls: Many are safe; some (e.g., open(), mmap()) can sleep. Know which.

Determinism vs. RTOS: Even a perfectly tuned PREEMPT_RT system is not as deterministic as a purpose-built RTOS (VxWorks, FreeRTOS, Zephyr). The trade-off is ecosystem: PREEMPT_RT gives you the full Linux userspace, drivers, networking stack, and tooling at the cost of some additional worst-case latency (typically 20–100 µs vs. 1–10 µs for a bare RTOS). For most industrial applications, this is an excellent trade-off.

PREEMPT_RT mainline status: As of Linux 6.12 (released November 2024), the core PREEMPT_RT infrastructure is fully merged into mainline Linux. The CONFIG_PREEMPT_RT option is now a first-class kernel build option. This was a major milestone — for years, PREEMPT_RT required a separate patchset applied on top of the mainline kernel.

#!/bin/bash
# Full RT system tuning checklist (x86, run as root)

# --- CPU Frequency ---
cpupower frequency-set -g performance
# Or directly: echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# --- Disable CPU C-states (sleep states add latency) ---
cpupower idle-set -D 0
# Or: for cpu in /sys/bus/cpu/drivers/processor/cpu*/power/pm_qos_resume_latency_us; do echo 0 > $cpu; done

# --- IRQ affinity: move all IRQs to CPU 0 ---
for irq in /proc/irq/[0-9]*/smp_affinity; do
    echo 1 > $irq 2>/dev/null
done

# --- RT throttle: disable (dangerous — only if you trust your tasks) ---
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us

# --- Verify isolation ---
cat /sys/devices/system/cpu/isolated
cat /sys/devices/system/cpu/nohz_full

# --- Measure baseline latency ---
cyclictest -m -p99 -t1 -i200 --duration=10s -a 2

# Target: max latency < 100 µs under hackbench load
# World-class: < 20 µs on well-supported hardware