Virtualization & KVM

To understand what hardware virtualization solves, you need to understand what made pure software virtualization difficult.

A CPU runs instructions at different privilege levels (rings). Ring 0 is kernel mode — the OS lives here and can execute any instruction. Ring 3 is user mode — applications run here. Privileged instructions (like lgdt, ltr, mov cr0, hlt) can only execute in ring 0. If a user-mode program tries them, the CPU raises a General Protection Fault.

A guest OS also expects to run in ring 0, doing privileged things. The classic solution was to run the guest at ring 1 (or ring 3 on x86) and intercept privileged instructions — a technique called trap-and-emulate. When a guest instruction triggers a fault because it can't run outside ring 0, the hypervisor (sitting in real ring 0) catches the trap, emulates the effect, and returns to the guest. Transparent!

The problem: x86 has ~17 instructions that change behavior silently depending on the privilege level instead of faulting. POPF (pop flags) is the most notorious: in ring 0 it modifies IF (the interrupt-enable flag), but in ring 1 it silently ignores the IF bit instead of faulting. The hypervisor never gets the trap, the guest never sees the expected behavior, chaos ensues.

This is Popek and Goldberg's 1974 virtualizability theorem: a machine is virtualizable if (and only if) every privileged instruction either faults or is sensitive (changes machine state) — so a hypervisor can always intercept the sensitive ones. Classic x86 violated this.

Solutions before hardware support:

• Full emulation (QEMU TCG): re-implement the entire CPU in software. Correct but 10–50× slower.
• Binary translation (VMware): scan guest code, rewrite offending instructions on the fly with safe equivalents. Fast (within 2–10×) but terrifyingly complex.
• Paravirtualization (Xen PV): modify the guest OS so it never uses the bad instructions — it calls hypervisor hypercalls instead. Fast, but requires porting every guest OS.

Hardware virtualization cuts this knot cleanly.

/* Classic 'sensitive but not privileged' instruction on x86 — SMSW
   In ring 3, reads CR0 into a register WITHOUT faulting,
   leaking whether protected mode is on. A hypervisor running
   the guest at ring 1 could never intercept this. */
uint16_t cr0_low;
asm volatile ("smsw %0" : "=r"(cr0_low));
/* Hardware VMX would instead cause a VM-exit here, letting KVM
   return a *virtualized* view of CR0 to the guest. */

Intel VT-x introduces VMX operation, a completely new CPU mode that sits alongside (not above) the ring 0/3 split.

Two new worlds:

• VMX root operation: where the hypervisor runs. Still has ring 0 and ring 3. This is where Linux + KVM live.
• VMX non-root operation: where the guest runs. Also has ring 0 and ring 3 — the guest OS runs at ring 0 just like it wants to — but the hardware automatically intercepts a configurable set of sensitive events and exits to the hypervisor.

The key data structure is the VMCS (Virtual Machine Control Structure) — a 4KB+ hardware-managed region of memory that holds:

1. Guest state: all registers (RIP, RSP, CR0, RFLAGS, segment descriptors, MSRs…) — the complete CPU state to restore when entering the guest.
2. Host state: a subset of registers to restore on VM exit.
3. VM execution controls: a bitmask of which events cause a VM exit. You can say "exit on every CPUID instruction" or "exit on writes to CR3" or "exit on I/O port access to port 0x3F8".
4. Exit info fields: when the CPU exits the guest, it fills these in — reason code, offending instruction, guest physical address that caused a fault, etc.

The four key instructions:

• VMXON: enables VMX operation (requires CR4.VMXE=1, checks IA32_FEATURE_CONTROL MSR).
• VMLAUNCH / VMRESUME: atomically saves host state into VMCS, restores guest state from VMCS, and enters VMX non-root at the guest's RIP. VMLAUNCH is for a fresh VMCS; VMRESUME is for re-entering after a VM exit.
• VMXOFF: disables VMX operation.
• VMREAD / VMWRITE: read/write fields in the current VMCS.

VM exit flow (the magic moment):

1. Guest executes a sensitive instruction (e.g., CPUID, or accesses a port, or triggers an EPT fault).
2. Hardware atomically: saves complete guest register state into VMCS guest-state area, restores host-state from VMCS host-state area, switches to VMX root ring 0, jumps to the hypervisor's VM-exit handler address (stored in the VMCS host RIP field).
3. Hypervisor reads VM-exit reason from VMCS, handles it, modifies guest state if needed.
4. VMRESUME sends control back to guest at the instruction after the offending one.

AMD's equivalent is SVM (Secure Virtual Machine), using VMCB instead of VMCS and VMRUN/VMEXIT instructions — conceptually identical, different register layouts.

/* Simplified: what a VM-exit handler skeleton looks like in KVM
   (actual code in arch/x86/kvm/vmx/vmx.c) */
static int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
{
    u32 exit_reason = vmx_get_exit_reason(vcpu); /* VMREAD exit reason */

    switch (exit_reason) {
    case EXIT_REASON_CPUID:
        return handle_cpuid(vcpu);   /* emulate CPUID */
    case EXIT_REASON_HLT:
        return handle_halt(vcpu);    /* guest halted, yield */
    case EXIT_REASON_EPT_VIOLATION:
        return handle_ept_violation(vcpu); /* page-fault in EPT */
    case EXIT_REASON_IO_INSTRUCTION:
        return handle_io(vcpu);      /* I/O port access */
    /* ... ~60 more exit reasons ... */
    default:
        vcpu_unimpl(vcpu, "unhandled exit 0x%x\n", exit_reason);
        return -EINVAL;
    }
}

KVM's architecture is its killer insight: don't write a hypervisor OS; teach Linux to be a hypervisor.

KVM ships as two kernel modules:

• kvm.ko — the generic, architecture-independent core.
• kvm-intel.ko or kvm-amd.ko — the architecture-specific layer that drives VMX or SVM.

When loaded, KVM registers a character device: /dev/kvm. Everything is built on three levels of ioctl():

Level 1 — System ioctls on /dev/kvm:

fd_kvm = open("/dev/kvm", O_RDWR);
fd_vm  = ioctl(fd_kvm, KVM_CREATE_VM, 0);   // create a VM

Level 2 — VM ioctls on the VM fd:

ioctl(fd_vm, KVM_SET_USER_MEMORY_REGION, &region); // map guest RAM
fd_vcpu = ioctl(fd_vm, KVM_CREATE_VCPU, 0);         // create a CPU

Level 3 — vCPU ioctls on the vCPU fd:

ioctl(fd_vcpu, KVM_SET_SREGS, &sregs);  // set segment registers
ioctl(fd_vcpu, KVM_RUN, 0);             // enter guest; blocks until VM exit

After KVM_RUN returns, userspace reads a shared memory page (struct kvm_run *) that tells it why the guest exited — did it do an HLT? An I/O port access? An MMIO access to an emulated device? This is the vCPU's communication channel between kernel (KVM) and userspace (QEMU).

What KVM handles in the kernel (fast path):

• EPT/NPT page-fault resolution (memory virtualization).
• APIC virtualization (Posted Interrupts, APICv).
• Many MSR reads/writes.
• Halt polling (spinning briefly before sleeping, to catch quick wakeups).

What KVM punts to userspace (QEMU's job):

• I/O port emulation (disk controllers, serial ports, RTC).
• MMIO emulation (PCI BARs, VirtIO queues).
• Firmware (BIOS/UEFI).
• Everything visual (VGA, GPU).

This split is why KVM is often called Type 1.5: the CPU virtualization is Type 1 (inside the kernel, running on bare metal), but device emulation is Type 2 (in a userspace process). You get Type 1 performance for the CPU-intensive path and Type 2 flexibility for device model hackability.

/* Minimal "Hello, KVM" — boots a 16-bit guest that writes
   to port 0x3F8 (serial) then halts.
   Adapted from Liz Rice's famous KVM demo. */
#include <fcntl.h>
#include <linux/kvm.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>

static const uint8_t code[] = {
    0xb0, 0x41,        /* mov al, 'A'  */
    0xe6, 0x3f,        /* out 0x3f, al  (simplified port) */
    0xf4,              /* hlt           */
};

int main(void) {
    int kvm = open("/dev/kvm", O_RDWR);
    int vm  = ioctl(kvm, KVM_CREATE_VM, 0);

    /* Map 4 KB of guest physical memory at GPA 0 */
    void *mem = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE,
                     MAP_SHARED|MAP_ANONYMOUS, -1, 0);
    memcpy(mem, code, sizeof(code));
    struct kvm_userspace_memory_region region = {
        .slot = 0, .guest_phys_addr = 0,
        .memory_size = 0x1000, .userspace_addr = (uint64_t)mem,
    };
    ioctl(vm, KVM_SET_USER_MEMORY_REGION, &region);

    int vcpu = ioctl(vm, KVM_CREATE_VCPU, 0);
    struct kvm_run *run = mmap(NULL,
        ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, 0),
        PROT_READ|PROT_WRITE, MAP_SHARED, vcpu, 0);

    /* Set up real-mode: CS:IP = 0:0 */
    struct kvm_regs regs = {.rip = 0, .rflags = 2};
    struct kvm_sregs sregs;
    ioctl(vcpu, KVM_GET_SREGS, &sregs);
    sregs.cs.selector = sregs.cs.base = 0;
    ioctl(vcpu, KVM_SET_SREGS, &sregs);
    ioctl(vcpu, KVM_SET_REGS, &regs);

    for (;;) {
        ioctl(vcpu, KVM_RUN, 0);
        if (run->exit_reason == KVM_EXIT_HLT) break;
        if (run->exit_reason == KVM_EXIT_IO)
            printf("Guest I/O: port=0x%x data=0x%x\n",
                   run->io.port, *(uint8_t *)((char *)run + run->io.data_offset));
    }
    puts("Guest halted.");
}

Memory virtualization is where things get wonderfully mind-bending. A guest OS thinks it owns physical memory. It sets up its own page tables, mapping guest virtual addresses (GVA) to what it believes are guest physical addresses (GPA). But those GPAs are not real — the hypervisor has mapped them to actual host physical addresses (HPA).

Without hardware support, this requires shadow page tables: the hypervisor secretly maintains a third set of page tables that directly maps GVA→HPA, and intercepts every guest CR3 write to rebuild the shadow. This is correct but horribly expensive — every guest TLB flush is a VM exit, and every new mapping synchronization costs dozens of kernel operations.

EPT (Intel) / NPT (AMD) solve this in hardware with a two-dimensional page table walk:

1. The guest has its own CR3 → page tables mapping GVA→GPA (normal guest page tables, untouched).
2. The hardware also has an EPT base pointer (in the VMCS) → EPT page tables mapping GPA→HPA.
3. On a memory access, the hardware walker combines both: walks the guest page table, and for every level of guest page table it touches, it also walks the EPT to translate the physical address of that page table entry. A full 4-level walk requires up to 24 memory accesses (5 EPT walks × 4 levels + 4 guest PT levels… but the TLB caches most of it).

An EPT violation (the hardware equivalent of a page fault in EPT) occurs when:

• The GPA has no EPT mapping yet (KVM adds it via mmap).
• A write hits a read-only EPT entry (used for copy-on-write, dirty tracking).
• A non-present guest page triggers the guest's own #PF handler while the EPT walk is still pending.

KVM handles EPT violations in the kernel fast-path when possible (just add a mapping) or punts to userspace for MMIO regions (a GPA with no backing physical memory maps to an emulated device).

Memory types and MTRR emulation: EPT entries also encode memory type (WB, UC, WT) — KVM must emulate the guest's MTRR/PAT registers and propagate the correct type into EPT, otherwise device DMA memory (which must be uncacheable) would be silently cached, causing data corruption.

Dirty tracking for live migration is elegant: mark all EPT entries read-only. The first write triggers an EPT violation. KVM records the page as dirty, marks it writable in EPT, resumes the guest. At migration time, dirty pages are the only ones that need re-transfer.

/* EPT entry format (64-bit), simplified */
/* Bits 0-2: R/W/X permission (separate execute permission!) */
/* Bits 3-5: memory type (0=UC, 6=WB for normal RAM)        */
/* Bits 12-51: host physical page frame number              */

#define EPT_R    (1ULL << 0)
#define EPT_W    (1ULL << 1)
#define EPT_X    (1ULL << 2)
#define EPT_MT_WB (6ULL << 3)  /* write-back memory type */
#define EPT_PFN(hpa) ((hpa) & ~0xFFFULL)

typedef uint64_t epte_t;

/* Create a leaf EPT entry mapping GPA -> HPA */
static epte_t make_ept_leaf(uint64_t hpa, bool writable, bool executable) {
    epte_t e = EPT_PFN(hpa) | EPT_R | EPT_MT_WB;
    if (writable)   e |= EPT_W;
    if (executable) e |= EPT_X;
    return e;
}
/* For dirty tracking (live migration): clear EPT_W, causing
   an EPT violation on the next guest write to this GPA. */
static epte_t ept_write_protect(epte_t e) { return e & ~EPT_W; }

A vCPU (virtual CPU) is a kernel thread from the host's perspective. KVM creates one kernel thread per vCPU; the Linux scheduler schedules vCPU threads like any other thread. This is the genius of the KVM design: you get CPU isolation, NUMA affinity, cgroups, and all the other scheduler features for free.

When a vCPU thread runs, it executes kvm_arch_vcpu_ioctl_run(), which calls into the architecture-specific vmx_vcpu_run() (for Intel), which executes VMLAUNCH/VMRESUME to enter the guest. The guest then runs directly on the physical CPU — executing native instructions — until something triggers a VM exit.

VM exit cost is the key performance metric. Each exit involves:

1. Hardware saves ~20–30 guest registers to VMCS (plus MSRs if configured).
2. Hardware restores host state (CR3, RSP, RIP to VM-exit handler, segments, some MSRs).
3. Software reads exit reason, dispatches handler.
4. Handler either handles it in kernel (fast, ~hundreds of ns) or returns to userspace (slow, ~microseconds — a full syscall round-trip).
5. VMRESUME saves host state, restores guest state, enters non-root.

For a workload that rarely triggers exits (e.g., pure compute), overhead is near zero. For I/O-heavy workloads, each I/O port access is potentially an exit → userspace → QEMU emulation → re-entry. This is why virtio (see next concept) matters so much.

Halt polling: When a guest vCPU executes HLT (waiting for an interrupt), KVM could immediately schedule another thread. But if the guest is just waiting for a network packet that will arrive in 50 microseconds, sleeping and waking is more expensive than spinning. KVM implements halt polling: spin for a configurable duration (/sys/module/kvm/parameters/halt_poll_ns, default 500,000 ns = 0.5 ms) before sleeping. This dramatically reduces latency for interactive workloads at the cost of wasted CPU.

Posted interrupts (Intel's APIC virtualization): delivering a virtual interrupt to a guest normally requires a VM exit so KVM can inject the interrupt. Posted interrupts let the hardware deliver the interrupt directly to the guest without a VM exit if the vCPU is running — a table of pending virtual interrupts (Posted-Interrupt Descriptor) is checked by the CPU on VMRESUME, costing almost nothing.

# Check VM exit statistics for a running QEMU/KVM VM (Linux host)
# KVM exposes per-vCPU counters via debugfs
pid=$(pgrep qemu-system-x86)
ls /sys/kernel/debug/kvm/
# Lists VMs like: 12345-0  (pid-vmfd)

# Per-vCPU exit reasons:
cat /sys/kernel/debug/kvm/12345-0/vcpu0/exits
# Output:
# exits          4823901
# halt_exits     103221
# io_exits       612443
# mmio_exits     5543
# signal_exits   18
# irq_window     40211
# ...

# High io_exits means lots of port I/O — consider virtio
# High mmio_exits means MMIO device access — also consider virtio

Hardware virtualization gives you near-native CPU performance, but I/O is still painful. A real NIC or disk controller was designed to be driven by a single OS that owns the hardware. When QEMU emulates, say, an Intel e1000 NIC, the guest's e1000 driver talks to it as if it were real hardware: the driver writes to I/O ports and MMIO registers, QEMU traps every single write, interprets it, does the real network send. Even with the hardware acceleration, you're paying for the emulation abstraction.

Paravirtualization says: instead of pretending to be hardware that was designed for a different world, design a new interface optimized for virtualization. The guest knows it is virtualized and cooperates. The hypervisor provides a clean, efficient API.

virtio is the Linux standard for paravirtual I/O, defined in VIRTIO spec. It defines:

1. Transport layer: how to discover and configure virtio devices. On modern systems, virtio-PCI — the device appears as a PCI device, the guest scans it normally, but the PCI config space has virtio-specific registers.

2. virtqueue: the core data structure — a split ring (or packed ring in virtio 1.1) consisting of:

   • Descriptor Table: an array of buffer descriptors (address, length, flags — next-chained, write-only…). A single transfer can chain multiple descriptors.
   • Available Ring (driver → device): the guest writes descriptor indices here to tell the device "I have a new request."
   • Used Ring (device → driver): the device writes back which descriptors it has consumed.
   • Memory-mapped in guest memory; host maps the same pages via EPT. No copying!

3. Device drivers: virtio-net, virtio-blk, virtio-scsi, virtio-gpu, virtio-balloon, virtio-rng, virtio-fs…

The key insight: virtqueue is a lock-free ring buffer in shared memory (guest RAM, mapped into host by EPT). The guest batches multiple I/O requests, writes them all to the available ring, then does a single kick (write to a notification register — one VM exit). The host processes the whole batch, writes completions to the used ring, and signals back (one injected interrupt). The ratio of VM exits to I/O operations plummets from 1:1 to roughly 1:N.

vhost: to avoid even the user-space round-trip for vhost-net, Linux moved the virtio-net backend into the kernel. The kernel vhost module processes virtqueue operations directly in a kernel thread, bypassing QEMU for the data path entirely. Network packets flow: guest virtio-net driver → virtqueue in guest RAM → vhost-net kernel thread → real socket → real NIC. Zero QEMU involvement.

vhost-user: moves the backend to userspace again but uses a Unix socket to share the virtqueue memory, allowing high-performance userspace backends (like DPDK-based Open vSwitch) to process virtio queues directly.

/* Simplified virtqueue 'kick' from a virtio-net driver perspective.
   After placing buffers in the available ring, driver notifies device. */

/* Guest driver adds buffer to available ring */
vq->avail->ring[vq->avail->idx % vq->num] = desc_idx;
smp_wmb(); /* ensure descriptor is visible before idx update */
vq->avail->idx++;
smp_wmb();

/* Check if device wants to be notified (optimization: device can
   suppress notifications by setting VRING_USED_F_NO_NOTIFY) */
if (!(vq->used->flags & VRING_USED_F_NO_NOTIFY)) {
    /* Single write to MMIO register — one VM exit for N requests */
    iowrite16(vq->index, vq->notify_addr);
}

/* On the host side (vhost kernel thread or QEMU), the notification
   triggers processing of ALL pending descriptors in the available ring.
   Host writes completions to used ring, then injects a single interrupt
   back to the guest (via KVM_SIGNAL_MSI or KVM Posted Interrupt). */

KVM is a kernel module — it provides CPU and memory virtualization but nothing else. It cannot boot a guest, present a display, emulate a disk controller, or parse a firmware file. QEMU fills this gap.

QEMU (Quick EMUlator) was originally a pure software emulator (TCG — Tiny Code Generator — a dynamic binary translator). When KVM is available, QEMU becomes the device model and KVM becomes the CPU engine. The pairing is:

• QEMU's job: open /dev/kvm, create VM and vCPUs, load firmware (UEFI/SeaBIOS), set up the guest memory map, emulate all devices (PCI bus, ACPI tables, virtio devices, AHCI/NVMe controllers, USB, display, NIC), handle I/O exits from KVM, inject interrupts, implement migration.
• KVM's job: VMLAUNCH/VMRESUME, EPT, interrupt injection, MSR virtualization, CPUID masking.

Startup sequence (simplified):

1. QEMU opens /dev/kvm, calls KVM_CREATE_VM.
2. QEMU mmaps guest RAM from the host, registers it with KVM_SET_USER_MEMORY_REGION. EPT is now set up.
3. QEMU loads firmware image into guest RAM at the right GPA (e.g., 0xFFFFF000 for BIOS).
4. QEMU creates vCPUs with KVM_CREATE_VCPU, sets initial register state (CS:IP pointing at firmware).
5. QEMU starts vCPU threads, calling KVM_RUN in a loop.
6. vCPU runs firmware → real mode → protected mode → boots guest OS.
7. Every I/O port or MMIO access the firmware or guest OS makes that hits an emulated device causes a KVM exit → QEMU handles it → KVM_RUN again.

Acceleration path for modern guests (a typical Linux guest with virtio drivers):

• Guest block I/O: guest virtio-blk driver → virtqueue kick (1 exit) → QEMU/vhost processes all queued requests → completion interrupt (1 interrupt injection). If vhost-blk is used, even QEMU is skipped.
• Guest network: virtio-net → vhost-net → host socket.
• Memory: EPT handles all page faults in kernel; QEMU only handles MMIO regions.

The result: a Linux guest with virtio drivers runs at roughly 95–98% of bare-metal performance for CPU-bound workloads, and close to wire speed for network and disk.

# Launch a full Linux VM with KVM acceleration and virtio devices
qemu-system-x86_64 \
  -enable-kvm \
  -cpu host \
  -m 2G \
  -smp 2 \
  -drive file=ubuntu.qcow2,format=qcow2,if=virtio \
  -netdev user,id=net0 -device virtio-net-pci,netdev=net0 \
  -nographic \
  -serial mon:stdio

# -enable-kvm: use KVM instead of TCG (software emulation)
# -cpu host: pass through host CPU flags (no masking)
# if=virtio: virtio-blk driver in guest, not emulated IDE
# virtio-net-pci: paravirtual NIC, not emulated e1000

# To see KVM in action, check /proc/cpuinfo inside the VM:
# The 'hypervisor' flag will be set, and CPUID leaf 0x40000000
# will return 'KVMKVMKVM' — KVM's hypervisor signature.

Interrupts are how hardware tells the CPU "something happened" — a packet arrived, a disk completed, a timer fired. In a VM, these interrupts must be virtual: the guest's interrupt controller (Local APIC + I/O APIC) is emulated, and the hypervisor must inject interrupts at the right time to the right vCPU.

The problem: you cannot simply inject an interrupt at any moment. The guest might have interrupts disabled (IF=0 in RFLAGS). The guest might be in a critical section. The virtual LAPIC has priority levels (TPR — Task Priority Register) — a high-TPR guest only wants high-priority interrupts.

Classic interrupt injection: KVM sets the interrupt-window exiting bit in the VMCS execution controls. This causes a VM exit at the first instruction where interrupts are enabled (IF=1). KVM can then set the VM-entry interruption-information field in the VMCS — the hardware will deliver the interrupt as if the real LAPIC had fired it, on the next VMRESUME.

APICv (Virtual APIC): Modern Intel CPUs have hardware acceleration for the APIC. The VMCS points to a Virtual-APIC Page — a shadow of the real LAPIC registers. Guest reads/writes to LAPIC MMIO are handled by hardware, without VM exits. Virtual interrupt delivery posts interrupts directly into the guest without a VM exit if the guest's TPR is low enough. The guest's EOI write (End of Interrupt) is also virtualized.

Posted Interrupts: For multi-vCPU VMs, delivering a virtual interrupt from device to the correct vCPU requires sending a real IPI (Inter-Processor Interrupt) to the physical CPU running that vCPU. Posted interrupts use a hardware mechanism where a physical IPI causes the CPU to check a Posted-Interrupt Descriptor, inject the virtual interrupt, and continue running the guest — no VM exit needed. This makes inter-VM networking (guest-to-guest via vhost) extremely efficient.

Timer virtualization: the guest's HPET, RTC, and LAPIC timer are all emulated. KVM programs real host timers (using hrtimer) and injects virtual timer interrupts at the correct frequency. The TSC offset (a VMCS field added to the hardware TSC on every guest read of RDTSC) makes the guest see a consistent virtual TSC that increases at the right rate regardless of vCPU scheduling jitter.

/* KVM: inject an IRQ into a guest vCPU (simplified from
   arch/x86/kvm/irq.c). This is used by QEMU to raise a
   virtual interrupt line (e.g., device IRQ 14 for IDE). */

struct kvm_interrupt irq = { .irq = 14 };  /* IRQ number */

/* Called from QEMU via ioctl(fd_vcpu, KVM_INTERRUPT, &irq) */
/* KVM then:
   1. Checks if vCPU has IF=1 and TPR allows this priority.
   2. If yes: sets VM-entry interruption-information in VMCS.
      Next VMRESUME delivers interrupt to guest as if LAPIC fired.
   3. If no: sets interrupt-window-exiting bit in VMCS controls.
      Hardware exits when guest opens interrupt window.
      KVM then injects the interrupt and clears the bit.
*/

/* With APICv, QEMU can instead write directly to the
   virtual APIC page (via KVM_SET_LAPIC), and the hardware
   handles delivery without any VM exit at all. */

Nested virtualization is the ability to run a hypervisor inside a VM — that is, run KVM inside a KVM guest. This is essential for:

• Cloud CI/CD: GitHub Actions runners on EC2 that need to run Docker (which now uses KVM for isolation).
• Hypervisor testing: develop and test a new KVM feature without rebooting your physical machine.
• Security research: run an untrusted hypervisor inside a safe outer VM.

The problem: a guest hypervisor (L1 guest) will try to execute VMXON, VMLAUNCH, and VMRESUME. In VMX non-root operation, these instructions cause a VM exit to the outer hypervisor (L0 = real KVM). The outer KVM must emulate the VMX machinery on behalf of L1.

How L0 KVM implements nesting (Intel):

1. L1 guest executes VMXON → VM exit to L0. L0 records that L1 is now in "virtual VMX root operation."
2. L1 executes VMWRITE to write to its "shadow VMCS" → L0 intercepts and maintains a software copy.
3. L1 executes VMLAUNCH to enter L2 (the nested guest) → L0 must merge L1's VMCS with L0's own VMCS to construct a merged VMCS that makes the hardware run L2 directly under L0's control.
4. L2 runs on real hardware (not double-emulated!) using a merged VMCS where:
   • Guest state = L2's registers.
   • Host state = L0's KVM handler.
   • Execution controls = union of L0's and L1's controls (conservative OR of intercepts).
5. When L2 causes a VM exit, L0 decides: can L0 handle this alone? (e.g., EPT fault for L2's memory) — handle it and VMRESUME back to L2. Or must L1 see it? — construct a fake VM exit in L1's VMCS and switch back to L1.

Performance: nested virtualization is slow (~2–3× penalty vs. non-nested) because the merged VMCS must be rebuilt on every L1↔L2 transition, and L2 VM exits that reach L0 add latency. But it's correct, and for development/testing workloads, it's invaluable.

Enable nested virt:

# Intel
modprobe kvm_intel nested=1
# Or permanently:
echo 'options kvm_intel nested=1' > /etc/modprobe.d/kvm-intel.conf

# Check:
cat /sys/module/kvm_intel/parameters/nested  # should print Y

The L1 guest sees vmx in /proc/cpuinfo flags, meaning it can run KVM itself.

# On your physical Linux host (L0):
modprobe kvm_intel nested=1

# Launch L1 VM, exposing VMX to the guest:
qemu-system-x86_64 -enable-kvm -cpu host,+vmx -m 4G \
  -drive file=ubuntu-l1.qcow2,if=virtio ...

# Inside L1 guest — verify VMX is visible:
grep vmx /proc/cpuinfo  # should show 'vmx' flag

# Inside L1, install KVM and launch L2:
sudo apt install qemu-kvm
qemu-system-x86_64 -enable-kvm -m 1G -drive file=tiny.qcow2 ...

# L2 is now running under nested KVM!
# L0's KVM merges VMCS entries from L1+L0 to run L2 on real HW.
# L2 VM exits go to L0; L0 decides whether to forward to L1.

The hypervisor is the security boundary. If a guest can escape the VM — read host memory, influence host kernel state, or interfere with other VMs — the entire cloud model collapses. KVM's attack surface is therefore of extreme interest to security researchers.

Meltdown (CVE-2018-3620/3646) exploited CPU speculative execution to read kernel memory from user mode. The fix — KPTI (Kernel Page-Table Isolation) — removes kernel mappings from user-mode page tables. For KVM specifically, L1TF (L1 Terminal Fault) was a variant: a guest could speculatively read host memory by crafting EPT entries with present=0 but a physical address pointing at host memory. The CPU speculatively fetches the L1 cache line before the EPT permission check. Fix: EPT poisoning (overwrite not-present EPT entries so the PFN bits don't point anywhere real) + core scheduling (don't let guest and host run on sibling hyperthreads).

Spectre (CVE-2018-3639, 5753, etc.) trained branch predictors to speculatively execute gadgets in the host kernel, leaking data via cache side-channels. Mitigations: retpoline (turn indirect branches into return-trampoline sequences that can't be mis-speculated to arbitrary gadgets), IBRS/IBPB (Indirect Branch Restricted Speculation, Indirect Branch Predictor Barrier — hardware MSRs that flush the BTB on guest/host switches), STIBP (prevent cross-hyperthread branch prediction injection).

The cost: every VM entry/exit now includes IBPB on some configurations, adding hundreds of nanoseconds. KVM has extensive perf and debugfs counters to measure this.

The broader attack surface:

• KVM emulation bugs: handle_xxx() functions that emulate guest instructions or devices. A bug here runs in kernel with full host privilege. CVE-2017-2596 (VPID fix), CVE-2019-3887 (nested VMX), CVE-2021-22543 (EPT page table management) are examples.
• QEMU device model: historically more vulnerable (it's complex C in userspace). The famous CVE-2015-3456 ("VENOM") was in QEMU's floppy controller emulation — buffer overflow, VM escape.
• Side channels: cache timing attacks (Flush+Reload, Prime+Probe) between VMs sharing a physical host. Mitigated by cache partitioning (Intel CAT) and careful scheduling.

As a KVM contributor, security fixes are high-priority and high-scrutiny. Every emulation path that touches guest-controlled data is a potential attack surface.

/* KVM L1TF mitigation: poison not-present EPT entries.
   Before the fix, a not-present EPT entry still had
   valid HPA bits the CPU would speculatively fetch.
   After: the PFN bits are set to an INVALID_PAGE sentinel
   that doesn't map to real host memory. */

#define INVALID_PAGE (~(hpa_t)0)    /* all 1s, not a real HPA */
#define REMOVED_SPTE  0x06ul        /* present=0 but poisoned PFN */

/* When an EPT entry is cleared (guest memory unmapped),
   instead of writing 0, write a poisoned value so that
   speculative L1 cache fetches go to an invalid address: */
static void kvm_set_mmio_spte(u64 *sptep)
{
    /* Mark as MMIO (special), not-present, but with
       a PFN that cannot be a real host page: */
    WRITE_ONCE(*sptep, REMOVED_SPTE);
    /* Hardware still sees present=0 (bit 0 clear),
       so a real access faults. But speculative fetch
       of the PFN bits hits only the poisoned sentinel. */
}

KVM is one of the most welcoming subsystems for new kernel contributors. Here is what you need to know to make your first patch land.

Maintainer structure:

• arch/x86/kvm/ — maintained by Paolo Bonzini (Red Hat) and Sean Christopherson (Google).
• virt/kvm/ — the arch-independent core.
• drivers/vhost/ — vhost maintainers track separately.
• Mailing list: kvm@vger.kernel.org. Subscribe via https://subspace.kernel.org.

Good first contributions:

1. Static analysis fixes: run make C=2 M=arch/x86/kvm to invoke sparse; fix real warnings.
2. Documentation: KVM's documentation in Documentation/virt/kvm/ is often behind the code.
3. CPUID feature exposure: new CPU flags need to be whitelisted in arch/x86/kvm/cpuid.c.
4. Emulation correctness: pick an x86 instruction that KVM emulates (arch/x86/kvm/emulate.c) and verify it against the Intel SDM.
5. Test cases: tools/testing/selftests/kvm/ has a growing KVM test suite. New test cases are always welcome.

Workflow:

1. Clone: git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git.
2. Build KVM: make defconfig && scripts/config -e KVM -e KVM_INTEL && make -j$(nproc) arch/x86/kvm/.
3. Run selftests: make -C tools/testing/selftests/kvm && ./tools/testing/selftests/kvm/x86_64/cpuid_test.
4. Write the patch, format it: git format-patch -1 --subject-prefix='PATCH'.
5. Check: ./scripts/checkpatch.pl --strict *.patch.
6. Find reviewers: ./scripts/get_maintainer.pl *.patch.
7. Send: git send-email --to kvm@vger.kernel.org *.patch.

What reviewers look for:

• Correctness against the Intel/AMD SDM (cite the manual section!).
• No new userspace ABI changes without extensive justification.
• Performance: does your change add VM exits? Add benchmarks.
• Security: did you handle malicious guest input? KVM treats all guest state as hostile.

# Build and run KVM selftests on your machine
cd /path/to/linux-source

# Build just the KVM module (fast)
make -j$(nproc) M=arch/x86/kvm modules

# Build the selftests
make -C tools/testing/selftests/kvm

# Run a specific test
./tools/testing/selftests/kvm/x86_64/vmx_set_nested_state_test

# Run all KVM selftests
./tools/testing/selftests/kvm/run_tests.sh

# Check a patch before sending
git format-patch -1
./scripts/checkpatch.pl --strict 0001-kvm-x86-your-change.patch
./scripts/get_maintainer.pl 0001-kvm-x86-your-change.patch
# Output: Paolo Bonzini <pbonzini@redhat.com>
#         Sean Christopherson <seanjc@google.com>
#         kvm@vger.kernel.org (open list:KERNEL VIRTUAL MACHINE...)