I/O: MMIO, Port I/O, DMA & Buses (PCIe)

A CPU executes instructions. Every instruction either computes something or moves data between registers and memory. The CPU has exactly one tool: the memory bus. A GPU, NIC, or SSD is not memory — it has registers, FIFOs, and DMA engines. So the question is: how do you shoehorn a device into the only thing a CPU knows how to talk to?

The answer goes back to the 1970s. Early microprocessors like the Intel 8080 solved this with a dedicated I/O bus: separate IN and OUT instructions, a separate 16-bit I/O address space (65,536 ports), and a separate IORQ# signal on the bus so devices knew the cycle was an I/O cycle, not a memory cycle. This kept devices out of the precious (and tiny, 64 KB on an 8-bit system) memory address space.

The x86 family inherited this with in/out instructions and a 16-bit I/O port space. But this model has a fatal flaw: it is architecturally special. Every architecture (ARM, RISC-V, MIPS, POWER) has a different opinion about I/O ports, and most modern ones have no separate I/O space at all — they never needed it because their address spaces were large enough from the start.

The clean solution — and the one that won — is memory-mapped I/O: assign the device's internal registers a range of physical addresses in the normal memory map. When the CPU reads or writes those addresses, the memory bus delivers the transaction to the device, not to DRAM. The device responds. No special instructions, no special signals — just load and store.

This works because physical address space is a resource the hardware platform partitions. The BIOS/UEFI firmware, guided by the PCIe specification, carves out ranges like 0xFEC00000 for the IOAPIC, 0xFEE00000 for the local APIC, and multi-megabyte windows for each PCIe device's registers. The CPU's memory controller (the 'uncore') knows which physical address ranges route to DRAM and which route to the PCIe root complex — and hence to devices.

/* x86 port I/O — reads one byte from I/O port 0x3F8 (COM1 UART) */
static inline uint8_t inb(uint16_t port) {
    uint8_t val;
    __asm__ volatile ("inb %1, %0" : "=a"(val) : "Nd"(port));
    return val;
}

/* MMIO — reads a 32-bit register from a device mapped at a virtual address.
 * In the kernel, ioremap() maps the physical BAR address to a kernel VA.
 * readl() is NOT a plain dereference — it issues a memory barrier. */
#include <asm/io.h>
void __iomem *bar = ioremap(phys_addr, size);
uint32_t status = readl(bar + 0x08);   /* offset 0x08 in BAR */
writel(0x1, bar + 0x04);               /* write 1 to offset 0x04 */
iounmap(bar);

When the OS boots, the BIOS/UEFI has already programmed each PCIe device's Base Address Registers (BARs) with a physical address window. The OS reads those BARs from PCIe config space (see the PCIe topology concept), then calls ioremap() (Linux) to establish a kernel virtual mapping for that physical range.

Now here is a subtlety that trips up nearly every beginner: you must not dereference MMIO pointers as if they were RAM. In C, *(volatile uint32_t*)ptr might work on your laptop, but it is undefined behavior for device registers for three reasons:

1. Ordering. The compiler and CPU are free to reorder memory accesses for performance. A device register write that arms a DMA transfer must not be reordered past the write that sets up the DMA descriptor. readl()/writel() and friends issue the appropriate memory barriers.

2. Caching. Device register memory must be mapped as uncacheable (UC) or write-combining (WC). If the CPU cached a device status register, it would never see the device updating it. The kernel's ioremap() sets the page-table attributes (the PAT bits on x86, the memory attributes in the ARM translation tables) to non-cacheable automatically.

3. Access size. Many devices require exactly 32-bit or 64-bit accesses to their registers. A byte-wide access to a 32-bit register is either broken or does something surprising. readl()/readq() enforce the correct access width.

The Linux MMIO API is beautifully explicit about this: readb/readw/readl/readq for 8/16/32/64-bit reads, writeb/writew/writel/writeq for writes. Each emits a memory barrier (mb()) to prevent CPU reordering. For performance-critical paths (like framebuffer writes to a GPU), iowrite32_rep() exists for burst writes, and ioremap_wc() enables write-combining for regions where you do not care about strict ordering.

On ARM and RISC-V, there is no port I/O at all — everything is MMIO. This is why modern kernel drivers prefer readl/writel over inl/outl: a driver written with MMIO APIs compiles and runs on x86, ARM, and RISC-V without modification.

/* Fragment from a real-ish PCI driver (simplified) */
static int mydev_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
    struct mydev *dev;
    int err;

    err = pci_enable_device(pdev);        /* power on, claim PCI resources */
    if (err) return err;

    err = pci_request_regions(pdev, "mydev"); /* reserve BAR address ranges */
    if (err) goto err_disable;

    /* BAR 0: 64-bit MMIO registers */
    dev->regs = pci_ioremap_bar(pdev, 0); /* ioremap BAR 0 */
    if (!dev->regs) { err = -ENOMEM; goto err_release; }

    /* Now safe to use readl/writel */
    uint32_t version = readl(dev->regs + REG_VERSION);
    pr_info("mydev: firmware version 0x%x\n", version);
    return 0;

err_release:
    pci_release_regions(pdev);
err_disable:
    pci_disable_device(pdev);
    return err;
}

The x86 architecture has a 16-bit I/O port address space accessed via the in and out family of instructions. This is entirely separate from the memory address space — outb(0x80, 0x61) does not touch RAM at all; it generates a bus cycle with the IORQ# signal asserted, and only devices that claim port 0x61 respond.

Classic x86 port assignments that you will see in old code and still functional in boot firmware:

• 0x20–0x21: 8259A PIC master (legacy interrupt controller)
• 0x40–0x43: 8253/8254 PIT (programmable interval timer)
• 0x3F8–0x3FF: COM1 serial UART
• 0xCF8–0xCFF: PCI configuration space access (the infamous CONFIG_ADDRESS / CONFIG_DATA ports)

That last one is critically important. The original PCI spec defined that the OS accesses device configuration space via two 32-bit ports: write the target (bus, device, function, register) as a 32-bit value to 0xCF8 (CONFIG_ADDRESS), then read or write 0xCFC (CONFIG_DATA). This is how BIOS/Linux enumerates PCI devices on x86 even today — port I/O is used not for data transfer but for device discovery on legacy systems.

PCIe introduced an alternative: the ECAM (Enhanced Configuration Access Mechanism), which maps all of PCIe config space into a 256 MB MMIO window. Modern Linux uses ECAM when available, falling back to CONFIG_ADDRESS/CONFIG_DATA for compatibility. On ARM and RISC-V systems, ECAM is the only mechanism.

In Linux kernel code, port I/O is wrapped by inb/inw/inl (read 8/16/32 bits) and outb/outw/outl (write). User-space programs on Linux can also use these after calling iopl(3) or ioperm(2) — this is how tools like setpci work.

/* x86 PCI config space access via legacy CONFIG_ADDRESS/CONFIG_DATA ports.
 * addr encodes: bit31=enable, bits30-24=0, bits23-16=bus,
 *               bits15-11=dev, bits10-8=func, bits7-2=reg, bits1-0=0 */
#define PCI_CONFIG_ADDRESS  0xCF8
#define PCI_CONFIG_DATA     0xCFC

static uint32_t pci_config_read(uint8_t bus, uint8_t dev,
                                uint8_t func, uint8_t reg) {
    uint32_t addr = (1u << 31)           /* enable bit */
                  | ((uint32_t)bus  << 16)
                  | ((uint32_t)dev  << 11)
                  | ((uint32_t)func <<  8)
                  | (reg & 0xFC);        /* dword-aligned */
    outl(addr, PCI_CONFIG_ADDRESS);
    return inl(PCI_CONFIG_DATA);
}

/* Check vendor ID of bus=0, dev=0, func=0 (usually the host bridge) */
/* uint32_t v = pci_config_read(0, 0, 0, 0); */
/* Vendor ID = v & 0xFFFF, Device ID = (v >> 16) & 0xFFFF */

Imagine the CPU wanting to receive 1 MB of data from a network card. The naive approach: the NIC signals an interrupt for every byte or packet; the CPU runs an interrupt handler; the handler reads a register; stores the byte to RAM; repeat. Even at a modest interrupt rate this burns 100% of CPU time on memory copies and interrupt overhead. For 10 Gbps networking, it is physically impossible.

The solution is Direct Memory Access: the CPU tells the device where in RAM to put the data, then steps aside. The device becomes a bus master — it can issue read and write transactions on the PCIe bus targeting system RAM, completely independently of the CPU. When it is done, it fires an interrupt. The CPU wakes up to find the data already in RAM.

The sequence is:

1. The CPU allocates a physically-contiguous (or scatter-gather) buffer in RAM. Crucially, it must be pinned — the OS must not swap it out or migrate it while the device is using it. In the kernel, dma_alloc_coherent() does this and returns both a virtual and a DMA address (the address the device will use).
2. The CPU programs the device's DMA engine with the DMA address, transfer length, and direction (device-to-memory or memory-to-device).
3. The device DMA engine issues PCIe Memory Write TLPs (Transaction Layer Packets) targeting the DMA address. The PCIe root complex routes these writes to the memory controller, which writes them to DRAM.
4. The device fires an interrupt (or sets a completion flag in the descriptor ring).
5. The CPU receives the interrupt, calls dma_sync_single_for_cpu() if necessary, and reads the data.

Cache coherency is the silent complexity here. Modern x86 CPUs maintain cache coherency for DMA automatically — if the CPU has the DMA buffer line in cache, the incoming DMA write invalidates that cache line (this is hardware-enforced via the cache coherency protocol on the ring bus/mesh). On some embedded ARM systems, DMA is not cache-coherent, and the driver must explicitly flush/invalidate cache lines. This is why the kernel has both dma_alloc_coherent() (always coherent, may use uncached memory) and dma_map_single() + dma_sync_* (streaming DMA, may need explicit sync).

Scatter-Gather DMA extends this: instead of one contiguous buffer, the device is given a list of (address, length) pairs. The device's DMA engine walks this list, assembling data from (or to) non-contiguous memory regions. This is essential for receiving OS network buffers, which are rarely physically contiguous.

/* Simplified coherent DMA allocation + device programming */
#include <linux/dma-mapping.h>

struct rx_buf {
    dma_addr_t dma_handle;   /* physical address the device sees */
    void      *virt_addr;    /* virtual address the CPU sees */
    size_t     size;
};

static int alloc_rx_buffer(struct device *dev, struct rx_buf *buf)
{
    buf->size = 4096;
    /* dma_alloc_coherent: physically contiguous, cache-coherent,
     * pinned, correct attributes set automatically */
    buf->virt_addr = dma_alloc_coherent(dev, buf->size,
                                        &buf->dma_handle, GFP_KERNEL);
    if (!buf->virt_addr)
        return -ENOMEM;

    /* Program device DMA engine (device-specific register writes) */
    writel(lower_32_bits(buf->dma_handle), dev_regs + RX_DMA_ADDR_LO);
    writel(upper_32_bits(buf->dma_handle), dev_regs + RX_DMA_ADDR_HI);
    writel(buf->size,                      dev_regs + RX_DMA_LEN);
    writel(DMA_START,                      dev_regs + RX_DMA_CTRL);
    return 0;
}

static void free_rx_buffer(struct device *dev, struct rx_buf *buf)
{
    dma_free_coherent(dev, buf->size, buf->virt_addr, buf->dma_handle);
}

Here is a terrifying thought: DMA gives a device arbitrary write access to any physical address in your system. If a malicious or buggy device writes to the wrong physical address, it can corrupt kernel memory, crash the system, or — with a carefully crafted write — escalate privileges. Every Thunderbolt/DMA attack works this way.

The IOMMU (Input-Output Memory Management Unit) is the hardware answer. Just as the CPU's MMU translates virtual addresses to physical addresses (and enforces page-level permissions), the IOMMU translates device DMA addresses to physical addresses and enforces permissions. The device sees an IOVA (I/O Virtual Address) that the IOMMU translates before the memory access hits DRAM.

The IOMMU sits between the PCIe root complex and the memory controller. It holds a per-device page table (the DMAR table on Intel, the SMMU on ARM). When a device issues a DMA write to IOVA 0x12345000, the IOMMU:

1. Looks up the device's page table entry for that IOVA.
2. Checks that the device has write permission to this mapping.
3. Translates to the real physical address.
4. Forwards the write to the memory controller.

If the device tries to write to an IOVA it was not given, the IOMMU generates a fault (logged as a kernel warning like DMAR: DRHD fault addr), and the transaction is dropped. The device cannot corrupt memory outside its allocated IOVA space.

In Linux, the IOMMU is managed by the iommu subsystem. The dma_alloc_coherent() call you saw above actually goes through the IOMMU — it allocates both a physical page and an IOVA mapping in the device's IOMMU domain. The dma_handle the driver gets back is the IOVA, not the raw physical address.

For virtualization, the IOMMU is indispensable. VFIO (Virtual Function I/O) lets you pass a real PCIe device directly into a VM (VFIO passthrough). The IOMMU ensures that the guest VM's DMA addresses stay within its own memory — the VM cannot DMA into the host kernel. VFIO is the mechanism behind GPU passthrough for gaming VMs and SR-IOV NICs.

On Intel, the IOMMU technology is called VT-d (Virtualization Technology for Directed I/O); on AMD it is AMD-Vi; on ARM it is the SMMU (System Memory Management Unit). Linux supports all three via the unified iommu API.

# Check if IOMMU is active on a Linux system
dmesg | grep -i iommu
# Intel: "DMAR: IOMMU enabled"
# AMD:   "AMD-Vi: AMD IOMMUv2 functionality enabled"
# ARM:   "arm-smmu ... probed"

# See IOMMU groups (devices that must be passed-through together)
for g in /sys/kernel/iommu_groups/*/devices/*; do
    echo "Group $(basename $(dirname $g)): $g"
done

# Force IOMMU strict mode (no batching, every map/unmap updates page tables)
# Add to kernel command line:
#   intel_iommu=on iommu=pt    (passthrough for trusted devices)
#   intel_iommu=on             (full IOMMU for all devices)

# Show IOMMU page faults (DMA violations):
dmesg | grep DMAR

PCI Express is a point-to-point serial interconnect, but it presents a tree topology to software, preserving the hierarchical bus/device/function addressing of classic PCI. Understanding the physical topology is essential for understanding latency, bandwidth, and why some devices can see each other's DMA and some cannot.

The tree has three kinds of nodes:

Root Complex (RC): The root of the PCIe tree, implemented as part of the CPU chip or a separate chipset. It interfaces the PCIe fabric to the CPU's memory subsystem (DRAM + caches). Every PCIe transaction originating from a device ultimately arrives at the root complex, which routes it to DRAM or back to the CPU. The root complex also contains (or is adjacent to) the IOMMU.

Switches: PCIe switches are the equivalent of Ethernet switches, but for TLPs. A switch has one upstream port (toward the root complex) and multiple downstream ports (toward endpoints or more switches). Traffic between two endpoints behind the same switch does not have to go up to the root complex — the switch can route it directly (peer-to-peer DMA). GPUSdirect exploits this: two GPUs behind the same switch can DMA to each other's memory without touching DRAM.

Endpoints: The actual devices — NICs, SSDs, GPUs, USB controllers. An endpoint has one upstream port and is the source or sink of data. A single physical device can present multiple functions (up to 8 per device slot), each with its own config space, BARs, and interrupt vectors. A combined NIC+iSCSI card might be function 0 (NIC) and function 1 (iSCSI HBA).

Addressing in PCIe software uses a 3-level hierarchy:

• Bus number (8 bits, 0–255): assigned by the OS during enumeration
• Device number (5 bits, 0–31): physical slot on the bus
• Function number (3 bits, 0–7): logical sub-device within a physical device

Together these form the BDF (Bus:Device.Function) — you see this in lspci output like 00:1f.3. The 256-byte (or 4096-byte for PCIe Extended Config Space) configuration space is per-function.

PCIe uses lanes for physical connectivity. A x1 link is one differential pair each direction (full-duplex). A x16 link is 16 pairs each direction — typical for GPUs. PCIe 3.0 x16 provides ~16 GB/s each direction. PCIe 5.0 doubles the bit rate again.

TLPs (Transaction Layer Packets) are the currency. The five you will see most:

• Memory Read Request / Completion with Data: the CPU reads a device register
• Memory Write: the CPU writes a device register, OR a device DMA-writes to RAM
• Configuration Read/Write: config space access
• Message: MSI/MSI-X interrupts, power management events

# lspci shows the PCIe tree. -t for tree view, -v for verbose.
lspci -tv
# Output:
# -[0000:00]-+-00.0  Intel Corporation... Host bridge
#             +-01.0-[01]--+-00.0  NVIDIA ... GP104 [GeForce GTX 1080]
#             +-1c.0-[02-05]---02:00.0  Samsung NVMe SSD
#             +-1f.3  Intel HDA audio

# Read a device's PCIe config space header:
lspci -s 01:00.0 -xxx

# Show PCIe speed and width:
lspci -s 01:00.0 -vvv | grep -E 'LnkSta|LnkCap'
# LnkCap: Port #0, Speed 8GT/s, Width x16
# LnkSta: Speed 8GT/s (ok), Width x16 (ok)

# Show all BARs (Base Address Registers = MMIO windows):
lspci -s 01:00.0 -v | grep 'Region'
# Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
# Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]

Every PCIe function has a configuration space — a standardized block of registers that describe the device's identity, capabilities, and resource requirements. The OS reads this space during boot to discover what devices are present and what resources they need. This is PCI enumeration.

The first 64 bytes of config space are the PCI header, standardized across all devices:

• Bytes 0x00–0x01: Vendor ID (e.g., 0x8086 = Intel, 0x10DE = NVIDIA)
• Bytes 0x02–0x03: Device ID (specific chip model)
• Bytes 0x04–0x05: Command register (enable bus mastering, enable MMIO, etc.)
• Bytes 0x06–0x07: Status register (capabilities pointer present, errors)
• Byte 0x08: Revision ID
• Bytes 0x09–0x0B: Class Code (0x020000 = Ethernet, 0x010802 = NVMe, 0x030000 = VGA)
• Bytes 0x10–0x27: BARs 0–5 (Base Address Registers)
• Byte 0x3C: Interrupt Line (legacy IRQ routing, mostly obsolete)
• Byte 0x3D: Interrupt Pin (which INTA#–INTD# wire, legacy)
• Byte 0x34: Capabilities Pointer (points to the linked list of capability structures)

BARs are the key resource negotiation mechanism. During enumeration:

1. The OS writes all-ones (0xFFFFFFFF) to a BAR register.
2. The device masks off the bits it does not implement, leaving only the size bits set.
3. The OS reads back and counts the trailing zeros to determine the required size.
4. The OS picks a suitable physical address range and writes it back to the BAR.
5. The OS writes the Command Register bit 1 (Memory Space Enable) to let the device respond to MMIO.

Capabilities form a linked list starting at the Capabilities Pointer. Each capability has a 1-byte ID and a 1-byte Next pointer. You will encounter: MSI (cap ID 0x05), MSI-X (0x11), PCIe (0x10), Power Management (0x01). The PCIe capability at offset 0x10 gives you the extended config space pointer and the link status registers you see in lspci -vvv.

In Linux, pci_read_config_dword(), pci_write_config_word() etc. wrap the ECAM or legacy port I/O accesses. The kernel's PCI core walks the bus tree at boot (see drivers/pci/probe.c), fills in struct pci_dev for each function, and registers them. Your driver's probe() function is called when the kernel finds a matching Vendor ID + Device ID.

/* How the kernel (simplified) sizes and assigns a BAR */
static resource_size_t pci_size_bar(struct pci_dev *dev, int bar)
{
    u32 orig, sz;

    pci_read_config_dword(dev, PCI_BASE_ADDRESS_0 + bar * 4, &orig);
    pci_write_config_dword(dev, PCI_BASE_ADDRESS_0 + bar * 4, ~0U);
    pci_read_config_dword(dev, PCI_BASE_ADDRESS_0 + bar * 4, &sz);
    pci_write_config_dword(dev, PCI_BASE_ADDRESS_0 + bar * 4, orig); /* restore */

    /* Clear flags bits (low 4 bits for memory BAR) */
    sz &= ~0xF;
    /* Size = lowest set bit, i.e. ~sz + 1 */
    return (~sz + 1);
    /* e.g. if sz reads back 0xFFF00000, size = 0x00100000 = 1 MB */
}

/* In a driver's probe(), after pci_enable_device(): */
resource_size_t bar_start = pci_resource_start(pdev, 0);
resource_size_t bar_len   = pci_resource_len(pdev, 0);
printf("BAR0: phys 0x%llx, size 0x%llx\n",
       (u64)bar_start, (u64)bar_len);

A device needs to tell the CPU "I am done" or "data arrived." The original PCI mechanism used wire-based interrupts: a dedicated sideband signal (INTA#–INTD#) pulled low by the device. The interrupt controller (8259A or IOAPIC) detected the line and forwarded an interrupt to a CPU. This is called INTx or legacy interrupt.

INTx has deep problems:

1. Sharing. Multiple devices share the same INTx line. Every device on the line must be polled in the ISR to find who fired. On a busy PCIe system with 20 devices sharing one line, this is a performance disaster.
2. Level-triggered. The line stays low until the device's status register is cleared. A driver that forgets to clear the register locks up the entire system (spurious interrupt storm).
3. No CPU targeting. All interrupts go to CPU 0. On a 64-core server, this makes one CPU handle all I/O interrupts while the rest are idle.

MSI (Message Signaled Interrupts), introduced in PCI 2.2, solves all three. Instead of a wire, an MSI interrupt is a memory write — the device DMA-writes a 32-bit data value to a 32-bit address (both programmed into the device's MSI capability registers by the OS). The memory write targets the local APIC's interrupt registers (0xFEE00000 on x86). The APIC interprets the write as an interrupt request for a specific vector on a specific CPU.

This is beautiful: interrupts become indistinguishable from DMA writes. There is no shared wire, no level-sensitivity, and the target CPU is encoded in the address bits — so the OS can steer interrupts to specific cores for NUMA locality.

MSI-X extends MSI to 2048 independent vectors per device (MSI supports only 32). Each MSI-X vector has its own (address, data) pair stored in an MSI-X table — a memory region within one of the device's BARs. A NIC with 64 RX queues can have 64 MSI-X vectors, one per queue, each steered to a different CPU core. This is the basis for RSS (Receive-Side Scaling) in high-performance networking.

In Linux, pci_alloc_irq_vectors(pdev, min, max, PCI_IRQ_MSIX) allocates MSI-X vectors; request_irq(pci_irq_vector(pdev, i), handler, ...) registers a handler per vector. The kernel's irq subsystem handles the IOMMU-side programming of the APIC addresses and the MSI-X table entries.

/* MSI-X setup in a driver probe() — simplified */
#include <linux/pci.h>
#include <linux/interrupt.h>

static irqreturn_t mydev_rx_irq(int irq, void *data)
{
    struct mydev_queue *q = data;
    /* process received packets in q */
    napi_schedule(&q->napi);
    return IRQ_HANDLED;
}

static int mydev_setup_interrupts(struct pci_dev *pdev, struct mydev *dev)
{
    int nvec, i, err;

    /* Request up to 16 MSI-X vectors, at least 1 */
    nvec = pci_alloc_irq_vectors(pdev, 1, 16, PCI_IRQ_MSIX | PCI_IRQ_MSI);
    if (nvec < 0) return nvec;
    dev->num_queues = nvec;

    for (i = 0; i < nvec; i++) {
        /* pci_irq_vector() returns the Linux IRQ number for vector i */
        err = request_irq(pci_irq_vector(pdev, i),
                          mydev_rx_irq, 0, "mydev-rx", &dev->queues[i]);
        if (err) goto err_free_irqs;

        /* Optionally: pin this IRQ to CPU i for NUMA locality */
        irq_set_affinity_hint(pci_irq_vector(pdev, i),
                              cpumask_of(i % num_online_cpus()));
    }
    return 0;
err_free_irqs:
    while (--i >= 0)
        free_irq(pci_irq_vector(pdev, i), &dev->queues[i]);
    pci_free_irq_vectors(pdev);
    return err;
}

Now that you understand MMIO, DMA, and MSI-X, you have all the pieces to understand how real high-performance I/O works in practice: descriptor rings (also called descriptor queues or rings).

A descriptor ring is a circular array of small fixed-size descriptor structures in DMA-accessible memory. Each descriptor tells the device about one I/O operation: the DMA address of the data buffer, its length, flags, and status. The ring has a head pointer (where the CPU adds new work) and a tail pointer (where the device picks up work and writes completions).

For a NIC receive path:

1. Driver initialization: The driver allocates the ring (say, 256 descriptors) and 256 receive buffers. It writes each buffer's DMA address into the corresponding descriptor and advances the tail pointer. It programs the ring's DMA address, size, and head/tail register addresses into the NIC via MMIO.
2. Packet arrives: The NIC picks the next descriptor at its internal head, DMA-writes the packet data to the buffer address in the descriptor, marks the descriptor as done (sets a done bit in the descriptor), and advances its internal head.
3. Interrupt (MSI-X): The NIC fires an MSI-X interrupt when N descriptors are done (or after a timeout).
4. Driver processing: The ISR (or NAPI poll function) walks descriptors from the driver's last known head, processing all with done=1. For each done descriptor, it passes the buffer to the network stack and allocates a fresh buffer for the next cycle.
5. Replenishment: The driver writes the new buffer DMA addresses into the descriptors it just consumed and advances the tail pointer register (via MMIO) to tell the NIC there are new descriptors available.

This ring design is why modern NICs and NVMe SSDs achieve millions of IOPS with almost zero CPU overhead. The CPU only touches the ring on interrupt; the device works independently. NVMe uses exactly this model: submission queues and completion queues are descriptor rings, and a 64-byte NVMe command is essentially a descriptor.

/* Simplified NVMe-like submission queue descriptor */
struct nvme_command {
    __u8  opcode;       /* 0x01 = Write, 0x02 = Read */
    __u8  flags;
    __u16 command_id;   /* used to match completions */
    __le32 nsid;        /* namespace ID */
    __le64 reserved;
    __le64 metadata;    /* metadata DMA address (if used) */
    __le64 prp1;        /* 1st physical region page (data DMA addr) */
    __le64 prp2;        /* 2nd PRP or PRP list pointer */
    /* ... command-specific DWORDs 10-15 ... */
    __le32 cdw10;       /* starting LBA low */
    __le32 cdw11;       /* starting LBA high */
    __le32 cdw12;       /* number of logical blocks - 1 */
    __le32 cdw13, cdw14, cdw15;
} __packed; /* 64 bytes total */

/* To submit: write descriptor to SQ[tail], then:
 * writel(new_tail, nvme_dev->sq_doorbell);
 * -- this one MMIO write kicks the NVMe controller */

Let us trace a single outbound network packet from send() in userspace all the way to electrons on the wire, naming every mechanism we have studied.

1. send() syscall → kernel copies data from userspace buffer into an sk_buff (socket buffer).
2. Network stack processes the sk_buff through TCP/IP layers, prepends headers.
3. Driver ndo_start_xmit() is called with the sk_buff. The driver calls dma_map_single() on the sk_buff's data region. The IOMMU allocates an IOVA and programs its page tables; the returned dma_addr is the IOVA.
4. The driver writes a TX descriptor into the NIC's transmit ring: {dma_addr, length, flags}. It then does one MMIO write to the NIC's TX tail doorbell register — this is the kick that tells the NIC new descriptors are ready.
5. NIC DMA engine sees the doorbell, picks up the descriptor, reads the packet data from RAM via a PCIe Memory Read Request TLP (IOVA → physical address via IOMMU → DRAM). The root complex sends back a Completion with Data TLP carrying the packet bytes.
6. NIC serializes the bytes onto the wire (Ethernet frame, 8b10b or 64b66b encoding at the PHY layer).
7. NIC writes a TX completion descriptor back to a completion ring in RAM (another DMA write). It then fires an MSI-X interrupt: a PCIe Memory Write TLP to 0xFEE[CPU]xxx — the local APIC's interrupt pending register. The targeted CPU receives an interrupt vector.
8. The CPU runs the TX completion ISR. It calls dma_unmap_single() (IOMMU unmaps the IOVA), frees the sk_buff, and updates the TX ring head pointer.

Every step in this sequence uses exactly one of: MMIO register write (doorbell), DMA (packet fetch, completion write), MSI-X (interrupt), or IOMMU (address translation + permission). The architecture is not accidental — it is a careful design that minimizes CPU involvement to two tiny critical-path windows: queueing the descriptor and processing the completion.