R0KernelPath
module 032 / 262

Interrupts, Exceptions & Traps

Interrupts are the nervous system of a computer — this module explains exactly how hardware events hijack the CPU, save its soul, and hand control to the kernel.

~18h focused study11 concepts

Overview

When you press a key, plug in a USB drive, or divide by zero, the CPU does not finish what it was doing and politely wait — it interrupts itself, saves its entire mental state, runs a small piece of code to handle the event, then resumes exactly where it left off. This mechanism — interrupts, exceptions, and traps — is the heartbeat of every real operating system.

Without interrupts, a CPU would have to constantly poll every device ("is the keyboard ready? is the disk ready? is the network ready?") in an infinite loop, wasting billions of cycles on nothing. Interrupts let the hardware tell the CPU when something needs attention, freeing it to do real work in between. This is why your OS can run dozens of processes, respond to the mouse, stream audio, and serve network packets simultaneously on what is fundamentally a sequential machine.

The same mechanism that handles hardware events also handles software errors (like page faults and division by zero) and deliberate software-to-kernel transitions (system calls). The CPU does not distinguish much between "a device screamed at me" and "my own instruction caused a fault" — both go through the same fundamental machinery: the Interrupt Descriptor Table (IDT) on x86, or the vector table on ARM/RISC-V.

Understanding this machinery is not optional for kernel work. Every system call, every page fault handler, every device driver interrupt handler, every signal delivery in Linux passes through this path. When you read a kernel oops, the register dump you see is the CPU state saved at interrupt entry. When you write a kernel module that handles IRQs, you are writing a function that runs at the tip of this mechanism. This module gives you the complete, accurate mental model — from the electrical signal on a CPU pin all the way to the kernel's do_IRQ() and back.

why this matters for the kernel

Interrupt handling is the first place the kernel and hardware truly merge into one system. Every contribution to a device driver, every system call implementation, every real-time patch, and every security hardening effort (Spectre/Meltdown mitigations, KPTI, SMEP/SMAP) lives in or directly around this code path. You cannot read arch/x86/kernel/irq.c, understand entry_64.S, write a character device driver, or comprehend how strace works without a solid model of how the CPU transitions between user space and kernel space through the interrupt mechanism.

Beyond Linux specifically, this topic is your entry point into real-time systems, hardware security (interrupt-based side channels, SMM attacks), and CPU microarchitecture. The subtleties — interrupt latency, nested interrupts, interrupt affinity, the APIC timer as the kernel's clock source, NMIs and machine-check exceptions — appear constantly in performance debugging, bug triage, and architecture discussions. Mastering this module means you can look at a /proc/interrupts output, a perf stat trace, or a kernel crash dump and understand what happened at the hardware level.

interactive visualization
→
→
→
→
→
→
CPUUSER
PC0x004013F8
EAX0x0000002A
EBX0x7FFD9C40
FLAGS0x00000206
IRQ 1 line ASSERTED
Kernel Stackempty
(no saved frame)
Interrupt Vector Table
0Timer0xC0011200
1Keyboard0xC0011A40
2Cascade0xC00113E0
3Serial0xC0011C80
ISR @ 0xC0011A40
0 · RUNNING

The CPU is executing the user program's instruction stream, advancing the PC each step. A device asserts its IRQ line — the keyboard, in this case.

Watch the context frame get pushed onto the stack, the vector table route IRQ 1 to its ISR, then the frame pop back — the CPU resumes exactly where it left off.

x86-64 Interrupt Lifecycle: Device to Handler and Back

rendering diagram…

A hardware interrupt travels from the device through the I/O APIC and Local APIC to the CPU, which atomically saves state and jumps through the IDT to the Linux entry stub. The stub builds pt_regs, calls do_IRQ, which runs the driver handler and writes EOI, then returns via iretq restoring all state.

The concepts

01What Is an Interrupt? The Core Idea

A CPU normally executes instructions in sequence: fetch, decode, execute, repeat. An interrupt is an asynchronous signal — from hardware, software, or the CPU itself — that causes this sequential flow to be suspended so a special handler can run.

Think of it like this: you are reading a book (executing instructions). Your phone rings (interrupt signal). You put your finger on the current line (save program counter and state), answer the call (run interrupt handler), then return to exactly where you were (restore state and resume).

The CPU does this at hardware speed — typically within a handful of clock cycles of detecting the signal. The key properties that make this work:

  1. Atomicity of state save: The CPU saves the minimum architectural state needed to resume later — at minimum the instruction pointer (RIP/PC) and status flags. Most architectures also save the stack pointer and privilege level.
  2. Handler address lookup: The CPU does not guess where to jump. It uses a table — the IDT on x86, the vector table on ARM — indexed by an interrupt number called the vector.
  3. Privilege escalation: If the interrupt hits while user code is running, the CPU switches to kernel mode (ring 0 on x86) as part of the same atomic operation. This is fundamental to OS security.

There are three distinct flavors, which get their own sections: hardware interrupts (async, from devices), exceptions (sync, from the CPU itself due to the current instruction), and traps (deliberate, often syscall or int 0x80).

# See live interrupt counts on Linux:
cat /proc/interrupts

# Watch them change in real time:
watch -n 1 cat /proc/interrupts

02Hardware Interrupts: From Wire to Vector

A hardware interrupt begins with a physical signal. A keyboard controller, network card, or timer chip asserts a line — historically the IRQ (Interrupt ReQuest) lines on the old 8259A PIC, and now the pins or MSI (Message Signaled Interrupts) on the APIC.

The legacy 8259A PIC

The original IBM PC had two cascaded Intel 8259A Programmable Interrupt Controllers, giving 15 usable IRQ lines (IRQ0=timer, IRQ1=keyboard, IRQ2=cascade, etc.). The PIC would signal the CPU via the INTR pin, and the CPU would INTA (interrupt acknowledge) back, causing the PIC to put the vector number on the data bus.

The APIC era

Modern x86 systems use the Advanced Programmable Interrupt Controller (APIC), which comes in two parts:

  • Local APIC (LAPIC): one per CPU core, built into the processor die. Receives interrupts and can generate inter-processor interrupts (IPIs). Also drives the per-CPU timer (APIC timer).
  • I/O APIC: one (or more) on the motherboard, receives interrupt signals from devices, and routes them to the correct LAPIC using a redirection table.

The flow for a device interrupt:

Device asserts interrupt line
  → I/O APIC receives signal
  → I/O APIC consults its Redirection Table (which vector? which CPU?)
  → I/O APIC sends interrupt message to target LAPIC
  → LAPIC checks interrupt priority (TPR/PPR registers)
  → LAPIC asserts INTR to the CPU core
  → CPU finishes current instruction (or accepts at instruction boundary)
  → CPU checks IF flag (Interrupt Flag in RFLAGS) — if 0, defers
  → CPU saves state, looks up IDT[vector], jumps to handler

Maskable vs Non-Maskable Interrupts (NMI)

Most hardware interrupts are maskable: software can tell the CPU to ignore them temporarily by clearing the IF flag (cli on x86, DAIF bits on ARM). This is essential for critical sections in the kernel.

NMIs (Non-Maskable Interrupts) bypass the IF flag entirely — they always fire. They are reserved for catastrophic hardware errors: ECC memory errors, watchdog timeouts, hardware debugging (NMI watchdog in Linux). NMI handlers must be extremely careful and cannot use most kernel services.

MSI and MSI-X

Modern PCIe devices mostly use Message Signaled Interrupts: instead of a dedicated wire, the device writes a memory transaction to a special LAPIC address. This eliminates wire sharing, enables multiple vectors per device, and allows direct routing to a specific CPU core — crucial for network card performance (RSS, multi-queue).

/* In Linux, an interrupt handler is registered like this */
#include <linux/interrupt.h>

static irqreturn_t my_handler(int irq, void *dev_id)
{
    /* Do minimal work here — we are in interrupt context!
       Cannot sleep, cannot call schedule(). */
    struct my_device *dev = dev_id;
    handle_data(dev);
    return IRQ_HANDLED;
}

/* In probe/init: */
ret = request_irq(irq_number, my_handler,
                  IRQF_SHARED, "my_device", dev);
if (ret)
    return ret;

/* On exit: */
free_irq(irq_number, dev);

03Exceptions: Faults, Traps, and Aborts

While hardware interrupts come from outside the CPU, exceptions are generated by the CPU in response to something that happened during instruction execution. The Intel manuals define three sub-types:

Faults

A fault is reported before the faulting instruction completes. The saved instruction pointer points to the instruction that caused the fault, so the handler can fix the condition and re-execute the instruction. This is essential: a page fault (#PF, vector 14) is a fault. The kernel's page fault handler can allocate a physical page, map it, and return — and the faulting instruction will re-execute successfully, transparent to the program.

Common faults: General Protection Fault (#GP, vector 13), Page Fault (#PF, vector 14), Stack-Segment Fault (#SS, vector 12), Segment Not Present (#NP, vector 11).

Traps

A trap is reported after the trapping instruction. The saved IP points to the next instruction. Used for things that should not re-execute: the int3 single-step breakpoint trap (#BP, vector 3) — you hit the breakpoint, the handler fires, and when it returns, execution continues with the instruction after the int3. The overflow trap (#OF, vector 4) behaves similarly.

Aborts

An abort is a severe, unrecoverable condition. The CPU cannot reliably report the exact instruction that caused the problem. The machine-check exception (#MC, vector 18) is an abort — triggered by uncorrectable hardware errors (bad RAM, CPU internal errors). The kernel's MCE handler logs what it can and typically kills the process or panics.

The Dual-Fault and Triple-Fault

If an exception fires while handling another exception and the combination is invalid, a double fault (#DF, vector 8) occurs. If that faults, the CPU undergoes a triple fault — it resets. Historically, triple faults were used as a reset mechanism (BOCHS, old emulators). The Linux kernel sets up a dedicated double-fault stack using the IST (Interrupt Stack Table) mechanism precisely to handle cases where the normal kernel stack has overflowed.

TypeIP saved atCan retry?Examples
Faultfaulting instrYes#PF, #GP, #SS
Trapnext instrNo#BP, #OF, syscall
AbortundefinedNo#MC, #DF
/* Trigger a deliberate page fault to observe the mechanism.
   Run this as a normal user — the OS will send SIGSEGV. */
#include <stdio.h>
#include <stdint.h>

int main(void) {
    volatile uint8_t *bad = (uint8_t *)0xDEADBEEF;
    printf("About to fault...\n");
    fflush(stdout);
    uint8_t val = *bad;  /* page fault #PF fires here */
    printf("Never reached: %u\n", val);
    return 0;
}
/* The kernel's page fault handler sees no valid VMA,
   delivers SIGSEGV to the process. */

04Traps as a System Call Mechanism

The third class — deliberate traps — is how user programs ask the kernel for services. This is the system call mechanism.

Historical: int 0x80

On 32-bit x86 Linux, programs invoked system calls using int 0x80 (software interrupt, vector 128). The instruction number went in eax, arguments in ebx, ecx, edx, esi, edi. This caused a full IDT lookup and privilege switch — slow by modern standards.

Modern: syscall / sysret

Intel and AMD introduced the syscall/sysret pair (and ARM has svc, RISC-V has ecall). These are faster because they bypass the IDT lookup: the target handler address is stored directly in a Model-Specific Register (LSTAR on x86-64), and the CPU jumps there in one cycle. On AMD64 Linux:

  1. User code loads syscall number into rax, arguments into rdi, rsi, rdx, r10, r8, r9.
  2. syscall fires: CPU switches to ring 0, saves rip into rcx, saves rflags into r11, loads rsp from IA32_LSTAR area, jumps to entry_SYSCALL_64 in arch/x86/entry/entry_64.S.
  3. The kernel saves remaining registers, dispatches via sys_call_table[rax].
  4. sysret restores rip from rcx, restores rflags from r11, switches to ring 3.

The speed difference matters: int 0x80 took ~100+ cycles; syscall takes ~20–40 cycles on a modern CPU (before Spectre mitigations added swapgs overhead).

sysenter / sysexit

Intel's alternative (32-bit) to int 0x80, also MSR-based. The Linux vDSO picks the fastest available mechanism at runtime via the __kernel_vsyscall page.

/* Raw x86-64 syscall in assembly — write(1, msg, 13) */
.section .data
msg: .ascii "Hello, world\n"

.section .text
.global _start
_start:
    mov  $1,   %rax   /* SYS_write = 1           */
    mov  $1,   %rdi   /* fd = stdout             */
    lea  msg(%rip), %rsi  /* buf = &msg          */
    mov  $13,  %rdx   /* count = 13             */
    syscall           /* enter kernel           */

    mov  $60,  %rax   /* SYS_exit = 60          */
    xor  %rdi, %rdi   /* status = 0             */
    syscall

05The Interrupt Descriptor Table (IDT)

The Interrupt Descriptor Table is x86's lookup table for interrupt/exception handlers. It lives in physical memory and its address is held in the IDTR register (loaded with lidt). There are 256 entries (vectors 0–255). The first 32 (0–31) are reserved by Intel for CPU exceptions; vectors 32–255 are available for hardware interrupts and software traps.

IDT Entry Structure (x86-64)

Each entry is a gate descriptor — 16 bytes on x86-64:

  • Offset (64-bit): the handler function address, split across three fields.
  • Segment Selector: must point to a code segment in the GDT (almost always kernel CS, __KERNEL_CS).
  • Gate Type: Interrupt Gate (clears IF atomically — disables further interrupts), Trap Gate (leaves IF unchanged), Task Gate (rarely used, for hardware task switching).
  • DPL (Descriptor Privilege Level): the minimum CPL required to invoke this gate from software (int N). Exception gates have DPL=0 (only kernel can call them implicitly via faults); int3 has DPL=3 so user-space debuggers can use it.
  • IST (Interrupt Stack Table) index: 0–7. Non-zero values select an alternate stack from the TSS (Task State Segment) for this handler. Used for #NMI, #DF, #MC to guarantee a known-good stack even if the kernel stack overflowed.
  • P (Present) bit: if 0, accessing this entry triggers a Segment-Not-Present fault.

Trap Gate vs Interrupt Gate

This is subtle and important. An Interrupt Gate automatically clears the IF flag on entry, preventing other maskable interrupts from firing during the handler. A Trap Gate leaves IF alone — further interrupts can arrive. Linux uses interrupt gates for hardware IRQ handlers (we do not want to be preempted mid-handler) and trap gates for system calls (it is fine for another interrupt to fire while we are deep in a syscall).

ARM and RISC-V equivalents

ARM uses a vector table — a table of instructions (or since ARMv8, addresses) at a fixed base address stored in the VBAR_EL1 register. There are 16 entries covering sync/async exceptions from each combination of current/lower exception level and AArch64/AArch32 mode.

RISC-V uses the mtvec/stvec CSR (Control and Status Register). It can be set to direct mode (one handler for everything, distinguished by mcause/scause) or vectored mode (base address + 4*cause — similar to ARM).

/* x86-64 IDT entry layout (simplified, from Linux arch/x86/include/asm/desc_defs.h) */
struct gate_struct {
    u16 offset_low;     /* handler address bits 0..15  */
    u16 segment;        /* kernel CS selector          */
    struct idt_bits {
        u16 ist   : 3;  /* IST stack index (0=none)    */
        u16 zero  : 5;
        u16 type  : 4;  /* 0xE=interrupt, 0xF=trap     */
        u16 dpl   : 2;  /* privilege level             */
        u16 p     : 1;  /* present                     */
    } bits;
    u16 offset_mid;     /* handler address bits 16..31 */
    u32 offset_hi;      /* handler address bits 32..63 */
    u32 reserved;
} __attribute__((packed));

/* Reading the IDTR register: */
struct { u16 limit; u64 base; } __attribute__((packed)) idtr;
asm volatile ("sidt %0" : "=m"(idtr));
printk("IDT base=0x%llx limit=%u\n", idtr.base, idtr.limit);

06How the CPU Saves State at Interrupt Entry

When an interrupt or exception fires, the CPU must save enough state to resume the interrupted code exactly as if nothing happened. The mechanism is precise and hardware-defined.

What x86-64 pushes automatically (the hardware frame)

The CPU pushes these registers onto the kernel stack (switching stacks if coming from user mode via the RSP0 field in the TSS):

[RSP after push]  SS       (user stack segment, if CPL change)
                  RSP      (user stack pointer, if CPL change)
                  RFLAGS   (flags, including old IF bit)
                  CS       (code segment — encodes old privilege level)
                  RIP      (return address: faulting instr or next instr)
[optionally]      Error Code (for #GP, #PF, #SS, #DF, etc.)

The error code is pushed by the CPU for specific exceptions. For #PF, it encodes whether the fault was a read/write/exec, and whether it was user/supervisor. Not all exceptions push an error code — Linux's entry stubs push a dummy zero for exceptions that do not, so the stack layout is always uniform.

What software must save (the software frame)

The CPU only saves the above. General-purpose registers (rax, rbx, rcx, rdx, rsi, rdi, rbp, r8–r15) are the handler's responsibility. Linux's entry_64.S immediately pushes all of them via PUSH_REGS macro, building a struct pt_regs on the stack — the full saved register state that kernel code can inspect and that ptrace/gdb can read.

struct pt_regs in Linux

This structure is the kernel's representation of a saved user-space context:

struct pt_regs {
    unsigned long r15, r14, r13, r12;
    unsigned long rbp, rbx;
    unsigned long r11, r10, r9, r8;
    unsigned long rax, rcx, rdx, rsi, rdi;
    unsigned long orig_rax;  /* syscall number */
    unsigned long rip, cs, eflags, rsp, ss;
};

This is what you see in a kernel oops — the register dump is show_regs() printing this structure.

FPU / SSE / AVX state

Floating-point and vector registers (XMM, YMM, ZMM) are NOT saved by hardware on interrupt entry. They are saved lazily by the kernel using the XSAVE/XRSTOR instructions, only when the kernel itself needs to use FPU state (rare in interrupt context) or when context-switching to a different process. This lazy save is a significant performance optimization.

/* Minimal x86-64 interrupt entry stub (conceptual, simplified from entry_64.S) */
.macro SAVE_REGS
    push %rdi
    push %rsi
    push %rdx
    push %rcx
    push %rax
    push %r8
    push %r9
    push %r10
    push %r11
    push %rbx
    push %rbp
    push %r12
    push %r13
    push %r14
    push %r15
.endm

my_exception_handler:
    /* At this point, hardware has already pushed:
       [SS, RSP (if ring change), RFLAGS, CS, RIP, error_code] */
    SAVE_REGS
    mov %rsp, %rdi      /* pass pt_regs pointer as first arg */
    call do_my_handler
    /* RESTORE_REGS, then iretq */

07The APIC in Depth: Priority, EOI, and IPIs

The APIC is not just a wire-to-vector translator — it is a programmable subsystem with priority arbitration, masking, delivery modes, and inter-processor communication.

Interrupt priority

The LAPIC implements TPR (Task Priority Register) and PPR (Processor Priority Register). Any interrupt with a priority <= TPR is held pending; only interrupts with priority > PPR actually interrupt the CPU. Linux mostly leaves TPR=0 (accept all), but some real-time and virtualization scenarios use it.

IRR, ISR, TMR registers

  • IRR (Interrupt Request Register): 256-bit bitmap of pending interrupts (received but not yet dispatched to the CPU).
  • ISR (In-Service Register): 256-bit bitmap of interrupts currently being handled.
  • TMR (Trigger Mode Register): whether each vector is level-triggered or edge-triggered.

For level-triggered interrupts, the device holds the interrupt line asserted until the driver clears it. For edge-triggered, the device pulses the line. MSI interrupts are always edge-triggered.

EOI (End of Interrupt)

After a hardware interrupt handler finishes, it must write to the LAPIC's EOI register (a memory-mapped register). This clears the corresponding bit in the ISR, allowing lower-priority interrupts to be served. Forgetting an EOI is a classic driver bug: all further interrupts of that priority or lower stop working silently.

In Linux, irq_exit() → ack_APIC_irq() → apic_eoi() handles this.

Inter-Processor Interrupts (IPIs)

IPIs are how CPU cores signal each other. The LAPIC has an ICR (Interrupt Command Register): write a vector and a destination (self, all, all-excluding-self, or a specific CPU), and the message is sent over the system bus to the target LAPIC.

Linux uses IPIs constantly:

  • RESCHEDULE_VECTOR: tell another CPU to call schedule().
  • CALL_FUNCTION_VECTOR: run a function on another CPU (smp_call_function).
  • TLB_FLUSH_VECTOR: tell other CPUs to invalidate their TLB after a page table change.
  • REBOOT_VECTOR: shut down other cores during reboot.

The APIC Timer

The LAPIC also contains a timer that can fire an interrupt at a programmable rate. Linux uses this as its per-CPU tick source (or in tickless/nohz mode, fires it one-shot for the next scheduled event). The timer vector on Linux is LOCAL_TIMER_VECTOR (0xef).

/* Read LAPIC timer current count directly (x86-64, Linux kernel context)
   LAPIC is memory-mapped at fix_to_virt(FIX_APIC_BASE) */
#include <asm/apic.h>

/* Read the LAPIC ID of the current CPU: */
u32 lapic_id = apic_read(APIC_ID);
printk("LAPIC ID: 0x%x\n", lapic_id >> 24);

/* Read current timer count: */
u32 tmr = apic_read(APIC_TMCCT);
printk("APIC timer count: %u\n", tmr);

/* Send an IPI to all other CPUs (Linux helper): */
apic->send_IPI_allbutself(RESCHEDULE_VECTOR);

08Interrupt Latency: From Signal to First Handler Instruction

Interrupt latency is the time between an interrupt being asserted and the first instruction of the handler executing. For real-time systems this is a critical parameter. For Linux kernel contributors, it determines the responsiveness of drivers and timers.

Sources of latency

  1. Instruction completion: The CPU finishes the current instruction before accepting interrupts (with exceptions for very long instructions like rep movs — these are interruptible at element boundaries on x86).
  2. Masking (cli): If interrupts are masked, the CPU queues the interrupt and only delivers it when sti is executed. Linux keeps interrupts masked for very short critical sections — typically measured in dozens of nanoseconds. Spinlocks held with interrupts disabled extend this.
  3. SMM (System Management Mode): The CPU can be hijacked by firmware SMM interrupts (triggered by IOSF/SCI), which are completely invisible to the OS. SMM latency spikes of 100µs–1ms are a real problem for audio, real-time, and latency-sensitive code.
  4. Microcode and microarchitectural events: Cache misses in the IDT lookup, pipeline stalls, and even Spectre/Meltdown mitigations (retpoline, IBRS, STIBP, KPTI page table switches) add latency.
  5. APIC delivery: Edge-triggered interrupts deliver in 1–3 cycles after the device asserts; level-triggered add a small delay for the bus arbitration.

Measuring latency

Linux provides cyclictest (from rt-tests) to measure interrupt and scheduling latency:

sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0

Typical results: a PREEMPT_RT kernel achieves <50µs worst-case latency on modern hardware; a vanilla kernel may spike to milliseconds.

PREEMPT_RT and latency

The mainline Linux kernel has a CONFIG_PREEMPT option that allows preemption in most kernel code, and CONFIG_PREEMPT_RT (now partially merged) converts spinlocks to RT mutexes and makes almost all interrupt handlers preemptible, achieving hard real-time latencies. Understanding interrupt latency is the foundation for understanding why these patches are necessary and how they work.

/* Measure interrupt-to-handler latency using perf events.
   Run in a shell: */
// Record timer interrupt latency:
// sudo perf stat -e irq:irq_handler_entry,irq:irq_handler_exit \
//      -a sleep 1

/* Or use ftrace to trace specific IRQ handlers: */
// echo 1 > /sys/kernel/tracing/events/irq/irq_handler_entry/enable
// echo 1 > /sys/kernel/tracing/events/irq/irq_handler_exit/enable
// cat /sys/kernel/tracing/trace

09The Full Interrupt Lifecycle in Linux: do_IRQ to Return

Let us trace the complete path of a hardware interrupt through the Linux kernel on x86-64, from CPU detection to handler return.

1. Hardware entry (entry_64.S)

When the interrupt fires, the CPU pushes the hardware frame, looks up IDT[vector], and jumps to the stub in entry_64.S. The stub pushes a dummy error code (for exceptions without one) and the vector number, then calls common_interrupt. common_interrupt calls SAVE_REGS, sets up the pt_regs pointer, and calls do_IRQ(regs).

2. do_IRQ() (arch/x86/kernel/irq.c)

__visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs) {
    struct pt_regs *old_regs = set_irq_regs(regs);
    struct irq_desc *desc;
    unsigned int irq = ~regs->orig_ax;  /* vector → IRQ number */

    irq_enter();  /* disable preemption, update idle stats */
    desc = irq_to_desc(irq);
    if (likely(desc))
        handle_irq(desc, regs);
    irq_exit();   /* re-enable preemption, run softirqs */
    set_irq_regs(old_regs);
    return 1;
}

3. irq_enter() and interrupt context

irq_enter() increments the preempt count (specifically the hardirq count bits), which marks the CPU as being in interrupt context. This prevents the scheduler from preempting, and prevents kernel code from sleeping (any attempt to sleep in interrupt context triggers a might_sleep() warning or BUG).

4. handle_irq() → driver's handler

The irq_desc structure contains a chain of irqaction structs, one per registered handler. Each handler is called in turn until one returns IRQ_HANDLED. For shared IRQs, all handlers are called.

5. irq_exit() and softirqs

irq_exit() decrements the hardirq count. If we are not in a nested interrupt and softirq work is pending, it calls invoke_softirq() — which runs the softirq handlers (network RX processing, timer callbacks, RCU work, tasklets). This is the bottom half mechanism: the interrupt handler (top half) does minimal work and defers heavier processing to softirqs.

6. Return via iretq

After all handlers complete, the kernel restores general-purpose registers from the stack (RESTORE_REGS), then executes iretq (interrupt return). The CPU atomically pops RIP, CS, RFLAGS, RSP, SS from the stack, switches back to the old privilege level (ring 3 if interrupted user space), and resumes the interrupted code.

/* Observe the interrupt handler call chain with ftrace function graph tracer */
/* Run as root: */

// cd /sys/kernel/tracing
// echo do_IRQ > set_graph_function
// echo function_graph > current_tracer
// echo 1 > tracing_on
// sleep 0.1
// echo 0 > tracing_on
// cat trace | head -50

/* Expected output shows the full call tree:
   do_IRQ()
     irq_enter()
     handle_irq()
       handle_fasteoi_irq()
         my_driver_handler()   <-- your IRQ handler
     irq_exit()
       __do_softirq()
         net_rx_action()       <-- network bottom half, etc. */

10Nested Interrupts, Preemption, and the Interrupt Stack

Nested interrupts

On x86-64, hardware interrupts can be nested: a higher-priority interrupt can fire while a lower-priority handler is running, if the handler did not disable interrupts. Linux's hardware interrupt handlers run with IF=0 (interrupts disabled, because the IDT uses interrupt gates). This means true hardware interrupt nesting in Linux is limited — NMIs can always nest, and local_bh_enable() re-enables softirqs but not hardware interrupts.

However, exception nesting is common and intentional: a page fault can occur inside a spinlock-protected kernel path; the page fault handler runs on top of the existing kernel stack.

The kernel stack and IST

By default, x86-64 Linux uses a per-CPU 16KB kernel stack. An interrupt entering from user space switches to this stack (via the TSS RSP0). An interrupt entering while already in kernel mode just pushes onto the existing kernel stack — so deep nesting could theoretically overflow it.

For critical exceptions, Linux uses the Interrupt Stack Table (IST): the TSS contains up to 7 alternate stacks (IST1–IST7). IDT entries can specify which IST to use. Linux assigns:

  • IST1: DOUBLEFAULT_STACK (for #DF)
  • IST2: NMI_STACK (for NMIs — these can interrupt any code including other handlers)
  • IST3: DEBUG_STACK (for #DB, #BP)
  • IST4: MCE_STACK (for #MC)

This guarantees that even if the kernel stack overflowed (causing the original fault), the double-fault handler still has a valid stack.

CONFIG_VMAP_STACK

Since Linux 4.9, kernel stacks can be allocated with vmalloc guard pages (CONFIG_VMAP_STACK), catching stack overflows as page faults rather than silent memory corruption. The page fault handler itself uses a per-CPU fallback stack for this case.

/* Check IST assignments in Linux source:
   arch/x86/include/asm/page_64_types.h defines:
   DOUBLEFAULT_STACK = 1
   NMI_STACK         = 2
   DEBUG_STACK       = 3
   MCE_STACK         = 4

   arch/x86/kernel/traps.c sets them:
*/
static const __initconst struct idt_data def_idts[] = {
    INTG(X86_TRAP_DE,    asm_exc_divide_error),
    ISTG(X86_TRAP_NMI,   asm_exc_nmi, IST_INDEX_NMI),
    INTG(X86_TRAP_BP,    asm_exc_int3),
    ISTG(X86_TRAP_DF,    asm_exc_double_fault, IST_INDEX_DF),
    /* ... */
};

11Security Implications: KPTI, SMEP, SMAP, and Interrupt Timing Attacks

The interrupt mechanism is at the intersection of performance and security. Several major vulnerability classes and mitigations live here.

Meltdown and KPTI (Kernel Page-Table Isolation)

Meltdown (CVE-2017-5754) exploited the fact that on pre-KPTI kernels, kernel virtual addresses were mapped in user-space page tables (just marked non-accessible). The CPU's speculative execution could read kernel memory before the permission check, with data leaking via cache timing.

The fix — KPTI — uses two page table sets per process: a minimal one for user mode (no kernel mappings except a tiny trampoline) and the full one for kernel mode. On every interrupt/syscall entry from user space, the CPU switches page tables (via CR3 write). On every return to user space, it switches back. This adds 100–200 cycles to every syscall and interrupt — the performance cost of Meltdown mitigation.

SMEP and SMAP

  • SMEP (Supervisor Mode Execution Prevention): Prevents the CPU from executing instructions at user-space addresses while in kernel mode. This defeats the classic exploit technique of placing shellcode in user space and jumping to it from a kernel overflow.
  • SMAP (Supervisor Mode Access Prevention): Prevents the CPU from reading or writing user-space memory while in kernel mode, unless AC (Alignment Check flag) is set. Kernel code that legitimately needs to access user memory uses copy_from_user()/copy_to_user(), which temporarily sets AC via stac/clac instructions.

Interrupt timing side channels

Interrupts can be used as a side channel. The IRQ affinity attack: if an attacker can observe which CPU processes a specific IRQ (via /proc/interrupts polling), they may infer which CPU is running a victim process. More subtly, precise interrupt timing has been used in cache timing attacks (Flush+Reload, Prime+Probe) to trigger predictable microarchitectural state changes.

/* Check KPTI status on a running Linux system */
// dmesg | grep -i kpti
// Kernel command line: cat /proc/cmdline | grep nopti

/* Check SMEP/SMAP CPU features: */
grep -m1 'flags' /proc/cpuinfo | tr ' ' '\n' | grep -E 'smep|smap'

/* Observe page table switch overhead using perf: */
// sudo perf stat -e dTLB-load-misses,iTLB-load-misses \
//      -a -- sleep 1

Key insights

  • ✦Interrupts, exceptions, and traps are the SAME mechanism at the hardware level — all three index the IDT (or ARM vector table) and cause the CPU to atomically save state and jump to a handler. The distinction is in *who* triggered them and *when* relative to the instruction boundary.
  • ✦The CPU saves only a minimal hardware frame (RIP, CS, RFLAGS, RSP, SS, and optional error code) — saving all general-purpose registers is the *kernel's* job, done in the assembly entry stubs before C code is called.
  • ✦Fault vs. Trap is not just terminology: a fault saves the address of the faulting instruction so it can be retried — this is the entire basis for demand paging (page faults can be fixed and retried transparently).
  • ✦An Interrupt Gate in the IDT clears the IF flag atomically on entry, preventing preemption by other hardware interrupts. This is why you cannot sleep in an interrupt handler: the CPU is in a fragile, non-preemptible state until irq_exit().
  • ✦The APIC EOI (End of Interrupt) write is mandatory for level-triggered interrupts — forgetting it silently stops all further interrupts at that priority, a notoriously confusing driver bug.
  • ✦KPTI (Meltdown mitigation) causes a full page table switch (CR3 write) on every syscall and interrupt entry from user space — this is the direct performance cost of a hardware speculative execution vulnerability, and it lives in the interrupt entry path.
  • ✦The IST (Interrupt Stack Table) mechanism exists specifically because a double fault or NMI can fire when the normal kernel stack has overflowed — a dedicated known-good stack is the only safe option.
  • ✦Linux's bottom-half (softirq) mechanism is triggered at the *end* of interrupt handling (in irq_exit()), not during it — this is how the kernel achieves low interrupt latency while still doing substantial work (TCP processing, etc.) in response to a single IRQ.

Common pitfalls

  • ⚠Sleeping in interrupt context: calling any function that can sleep (kmalloc with GFP_KERNEL, mutex_lock, copy_from_user, schedule) inside an interrupt handler or softirq will trigger a 'BUG: sleeping function called from invalid context' — use GFP_ATOMIC for allocations, and defer blocking work to a workqueue.
  • ⚠Forgetting EOI for level-triggered interrupts: if your driver handles a level-triggered IRQ but does not clear the device's interrupt status register before returning, the I/O APIC will immediately re-fire the interrupt, causing an infinite loop (interrupt storm).
  • ⚠Confusing IRQ number with vector number: on x86, the vector is what the CPU uses (0–255, in the IDT), while the IRQ number is the Linux virtual IRQ (which may be remapped by the interrupt controller). request_irq() takes an IRQ number; the IDT operates on vectors.
  • ⚠Accessing user-space memory directly in interrupt context: user-space pointers are not valid in interrupt context (the process may not be current, and the page tables may not be set up for it). Always use the proper put_user/get_user or copy_to/from_user APIs in process context.
  • ⚠Assuming interrupts are always enabled: code that runs with interrupts disabled must be extremely short. Holding interrupts off for too long increases interrupt latency and can cause missed timer ticks, dropped packets, and audio glitches — always measure with cyclictest or ftrace.
  • ⚠Mistaking 'not sleeping' for 'interrupt safe': a spinlock with interrupts enabled is NOT safe if the spinlock can also be acquired from an interrupt handler — you must use spin_lock_irqsave/spin_lock_irqrestore to also disable local interrupts.
  • ⚠Ignoring IRQF_SHARED semantics: when multiple devices share an IRQ line and IRQF_SHARED is set, your handler WILL be called for other devices' interrupts too. Always check your own device's status register and return IRQ_NONE if the interrupt is not yours.

Practice

  1. 01
    hard

    Write a minimal Linux kernel module that registers a handler for a software-triggered interrupt using `int3` (vector 3). Use `register_die_notifier()` to intercept `#BP` exceptions, print the value of RIP from the pt_regs, and resume execution. Test it by inserting the module and running a small C program that executes `asm("int3")`. Verify the kernel log shows your message and the program does not crash (it receives SIGTRAP by default — install a SIGTRAP handler in your test program).

  2. 02
    easy

    Read `/proc/interrupts` on your Linux machine. Write a Python or shell script that samples it every 100ms for 10 seconds and reports: (a) the top 5 IRQs by total count, (b) the IRQ with the highest rate of change during the sampling period, and (c) which CPUs are handling each of the top IRQs (unbalanced vs balanced distribution). Explain what each of the top IRQs does.

  3. 03
    medium

    Use `ftrace` to trace the full call graph of `do_IRQ` for the local timer interrupt (vector 0xef on x86) for 100ms. Identify: (a) how many timer interrupts fired, (b) what softirqs ran in `irq_exit()`, and (c) the average duration of the handler in microseconds. Then repeat with `CONFIG_HZ=250` vs `CONFIG_HZ=1000` kernel builds if available (or just reason from the trace what would change).

  4. 04
    hard

    Write a bare-metal x86-64 program (runs without an OS, e.g. in QEMU with `-kernel`) that sets up a minimal IDT with a handler for vector 32 (first available hardware IRQ), programs the legacy 8259A PIC to route IRQ0 (timer, 18.2Hz) to vector 32, enables interrupts with `sti`, and counts 10 timer ticks in the handler before halting. Print the count to the serial port (port 0x3f8).

  5. 05
    medium

    Measure the overhead of KPTI on your system. Write a microbenchmark in C that calls `getpid()` (a very cheap syscall) in a tight loop for 10 million iterations, measuring total time with `clock_gettime(CLOCK_MONOTONIC)`. Run it normally, then with `nopti` on the kernel command line (if you can reboot with that option or use a VM). Compare the per-call latency. If you cannot disable KPTI, compare `getpid()` (crosses the user/kernel boundary) vs `clock_gettime(CLOCK_REALTIME)` via the vDSO (does NOT cross the boundary) to see the savings the vDSO provides.

Resources

📄Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3A: System Programming Guide
docsfree

The authoritative source for IDT structure, gate descriptors, exception definitions, APIC operation, and IST. Chapters 6 (Interrupt and Exception Handling) and 10 (Advanced Programmable Interrupt Controller) are required reading.

📄Linux Kernel Source: arch/x86/kernel/traps.c and arch/x86/entry/entry_64.S
docsfree

Bootlin's cross-referenced kernel source browser — read the actual interrupt entry path. Searching for IDTENTRY, do_IRQ, and SAVE_REGS gives the complete picture.

📕Understanding the Linux Kernel, 3rd Edition
bookpaidO'Reilly, by Daniel P. Bovet and Marco Cesati. Chapters 4 (Interrupts and Exceptions) and 11 (Timing Measurements) are exhaustive.

The deepest book-length treatment of the kernel interrupt path; covers the 2.6 kernel but the fundamentals are unchanged.

✎LWN.net: A thorough introduction to the Linux interrupt subsystem
articlefree

LWN's multi-part series on the modern genirq subsystem — covers irq_domain, irq_desc, and the interrupt chip abstraction used by all modern drivers.

📄OSDev Wiki: Interrupts
docsfree

Practical bare-metal perspective on setting up IDT, PIC, and APIC from scratch — essential for understanding what the kernel abstracts away.

📕Operating Systems: Three Easy Pieces — Chapter on Interrupts
bookfree

Free OS textbook; the virtualization section explains interrupt-based I/O vs polling with clarity ideal for beginners.

📄Kernel Newbies: Interrupt Handling
docsfree

Practical guide to writing interrupt handlers in kernel modules with worked examples.

✎MESI Blog: How x86 Interrupts Work
articlefree

LWN article on the IRQ subsystem rewrite (genirq) explaining the motivation and design of the modern Linux IRQ abstraction.

🔧rt-tests / cyclictest
toolfree

The standard tool for measuring interrupt and scheduling latency on Linux — indispensable for real-time and driver work.

📜Meltdown paper (Lipp et al., 2018)
paperfree

Original Meltdown paper — explains exactly why KPTI was necessary and how interrupt entry was exploited. Reading this makes KPTI's overhead viscerally understandable.

Checkpoints — you can now…

  • ✓You can explain the difference between a fault, a trap, and an abort — including which instruction the saved RIP points to in each case, and why that matters for page fault handling.
  • ✓You can describe the complete path of a hardware interrupt: from device pin assertion, through the I/O APIC redirection table, to the Local APIC, to the CPU, to the IDT lookup, to the Linux entry stub, to do_IRQ, to the driver handler, to EOI, to iretq.
  • ✓You can read and interpret the fields of an x86-64 IDT gate descriptor — offset, segment selector, gate type, DPL, IST index, and present bit — and explain what each field controls.
  • ✓You can explain why kernel interrupt handlers must not sleep, what 'interrupt context' means in terms of the preempt count, and how to defer blocking work to a workqueue.
  • ✓You can describe what KPTI does, why it was needed (Meltdown), and roughly how much overhead it adds to syscall/interrupt entry paths.
  • ✓You can set up and use ftrace to observe the interrupt handler call graph on a running Linux system, and interpret the output to identify top-half vs bottom-half (softirq) work.
  • ✓You can write a minimal Linux kernel module that registers an IRQ handler using request_irq(), handles the interrupt correctly (checking if it is yours, writing EOI via return value, not sleeping), and frees it on module exit.