The Block Layer & I/O Scheduling

Linux classifies I/O devices into two broad categories: character devices (keyboards, serial ports, audio — byte-at-a-time, no seeking) and block devices (HDDs, SSDs, SD cards — random-access, sector-granular). Block devices expose their storage as a flat array of fixed-size sectors (traditionally 512 bytes, now 4096 bytes on modern drives — the "Advanced Format" standard). Everything above the block layer — filesystems, swap, raw device access — works in terms of logical blocks, which are multiples of sectors chosen by the filesystem (typically 4 KB = 8 sectors).

The kernel represents each block device as a struct gendisk (generic disk) and each logical partition as a struct block_device. The gendisk holds the disk name, capacity, and — critically — a pointer to the request queue (struct request_queue). Every I/O operation is ultimately expressed as a request targeting a specific sector range on a specific gendisk.

Why abstract at all? Because the caller (ext4, btrfs, the page cache) should not have to know whether it is writing to a spinning HDD (where seeks are expensive, so adjacent requests should be sorted), an NVMe SSD (where reordering is pointless but parallelism is everything), or a network-backed block device like iSCSI. The block layer is that abstraction boundary.

/* Inspect a block device from user space */
#include <fcntl.h>
#include <sys/ioctl.h>
#include <linux/fs.h>
#include <stdio.h>
#include <stdint.h>

int main(void) {
    int fd = open("/dev/sda", O_RDONLY);
    if (fd < 0) { perror("open"); return 1; }

    uint64_t size_bytes;
    /* BLKGETSIZE64: total device size in bytes */
    if (ioctl(fd, BLKGETSIZE64, &size_bytes) == 0)
        printf("Device size: %llu bytes (%.1f GiB)\n",
               (unsigned long long)size_bytes,
               size_bytes / (1024.0*1024.0*1024.0));

    int sector_size;
    /* BLKSSZGET: physical sector size */
    if (ioctl(fd, BLKSSZGET, &sector_size) == 0)
        printf("Sector size: %d bytes\n", sector_size);

    close(fd);
    return 0;
}

The bio (Block I/O) structure (include/linux/blk_types.h) is the atom of the block layer. It describes a single I/O operation: a list of memory buffers (pages) to be read from or written to a contiguous range of sectors on a specific block device.

Think of a bio as a scatter-gather descriptor: the data on disk is always contiguous (a range of sectors), but the memory holding that data can be scattered across many non-contiguous pages. This is essential because the page cache may have assembled a 64 KB read from pages that happen to be all over physical RAM.

Key fields of struct bio:

• bi_bdev: which block device.
• bi_iter.bi_sector: starting sector on the device (64-bit, so devices > 2 TB work fine).
• bi_iter.bi_size: total bytes remaining.
• bi_opf: operation flags — REQ_OP_READ, REQ_OP_WRITE, REQ_OP_FLUSH, REQ_OP_DISCARD, etc.
• bi_io_vec / bi_vcnt: the scatter-gather list, an array of struct bio_vec {page, offset, len} entries.
• bi_end_io: completion callback, called when the I/O finishes.
• bi_private: opaque pointer for the submitter.
• bi_next: bios can be chained into a list.

A critical design decision: bios are not requests. A bio is the generic description of what data to move. A request (covered next) is the scheduling unit. Multiple bios can be merged into a single request if their sector ranges are adjacent — this is called bio merging and is a major source of throughput gain on HDDs.

The lifecycle: caller (usually the page cache or a filesystem) allocates a bio via bio_alloc(), fills in the fields, adds pages via bio_add_page(), and submits it via submit_bio(). The block layer takes it from there.

/* Kernel-space: submit a simple single-page read bio.
 * (Simplified from a real device mapper or filesystem path)
 * Requires: kernel module context, a valid block_device *bdev.
 */
#include <linux/bio.h>
#include <linux/blkdev.h>

static void my_bio_end_io(struct bio *bio)
{
    if (bio->bi_status)
        pr_err("I/O failed: %d\n", bio->bi_status);
    else
        pr_info("I/O complete, %u bytes\n",
                bio->bi_iter.bi_size == 0 ?
                /* size is decremented as I/O completes */
                4096 : 0);
    bio_put(bio); /* release reference */
}

int do_single_page_read(struct block_device *bdev,
                        sector_t sector,
                        struct page *page)
{
    struct bio *bio;

    /* Allocate bio; 1 = max number of bio_vec segments */
    bio = bio_alloc(GFP_KERNEL, 1);
    if (!bio)
        return -ENOMEM;

    bio_set_dev(bio, bdev);           /* which device    */
    bio->bi_iter.bi_sector = sector;  /* starting sector */
    bio->bi_opf = REQ_OP_READ;        /* it's a read     */
    bio->bi_end_io = my_bio_end_io;   /* completion cb   */

    /* Add our page: offset 0, 4096 bytes (one sector block) */
    bio_add_page(bio, page, 4096, 0);

    submit_bio(bio);  /* hand off to block layer */
    return 0;
}

A struct request is the scheduling unit: what the I/O scheduler sees, manipulates, and dispatches to the driver. Where a bio says "move this data", a request says "here is a batch of related I/Os I'd like you to execute, and I have thought about their order."

The conversion from bio to request happens in blk_mq_make_request() (the blk-mq path). Small, adjacent bios are merged: if bio B starts exactly where bio A ends, they become one request, halving the overhead on the driver and hardware. Two kinds of merges exist:

• Front merge: B's last sector + 1 == A's first sector. B gets prepended.
• Back merge (more common): A's last sector + 1 == B's first sector. B gets appended.

A struct request contains:

• q: the request_queue it belongs to.
• __sector: first sector.
• __data_len: total byte count.
• bio / biotail: the chain of bios this request aggregates.
• rq_flags: per-request flags (e.g., RQF_SORTED, RQF_SOFTBARRIER).
• cmd_type / cmd_flags: the operation and I/O priority.
• Scheduler-private data: at the end of the struct, each scheduler can embed its own bookkeeping.

The request_queue (struct request_queue) holds:

• The MQ dispatch queues.
• Queue limits (struct queue_limits): max sectors per request, max segments, alignment, etc. — these are hardware constraints the layer enforces.
• A pointer to the scheduler (elevator_type).
• The hardware dispatch function (make_request_fn, now replaced by blk-mq's queue_rq).

Understanding queue_limits is important for driver writers: if your hardware can only DMA 64 KB at a time, you set max_sectors accordingly, and the block layer will never hand you a request larger than that.

/* Inspect the queue limits of a block device from shell */
/* These files live in /sys/block/<dev>/queue/ */

$ cat /sys/block/sda/queue/max_sectors_kb
512

$ cat /sys/block/sda/queue/max_segments
128

$ cat /sys/block/sda/queue/physical_block_size
512

$ cat /sys/block/sda/queue/logical_block_size
512

$ cat /sys/block/nvme0n1/queue/max_sectors_kb
2048

$ cat /sys/block/nvme0n1/queue/nr_requests
1023

/* nr_requests: depth of the software staging queue per hw queue */

The original Linux block layer (circa 1994 — 2013) used a single lock-protected request queue per device. All CPUs contended on that lock to insert and remove requests. This worked fine when a spinning HDD could handle ~200 IOPS (nobody noticed the lock overhead), but by 2013, NVMe SSDs were delivering hundreds of thousands of IOPS and appeared with PCIe lanes that could sustain millions of IOPS on multi-queue hardware. The old design became the bottleneck: benchmarks showed that a high-end NVMe SSD could do 800K IOPS in isolation, but the kernel could only deliver ~400K because CPUs were stalling on a single spinlock.

Jens Axboe (the block layer maintainer) introduced blk-mq (merged in Linux 3.13, 2013). The key insight: hardware has multiple independent queues; software should too.

The blk-mq design has two tiers:

1. Software Staging Queues (per-CPU): One blk_mq_ctx per CPU. When a CPU submits a bio, it is converted to a request and placed in that CPU's local queue — no global lock needed. This is cache-hot and contention-free.
2. Hardware Dispatch Queues (per HW queue): One blk_mq_hw_ctx per hardware queue on the device (NVMe might have 32; an HDD has 1). Requests are flushed from software queues to hardware queues, potentially by any CPU.

The mapping between CPU software queues and HW queues is configurable: on a 64-core machine with a 32-queue NVMe, two CPUs share each HW queue. On an HDD with one queue, all CPUs share that single HW queue (but the per-CPU staging queues still eliminate the hot path lock).

This architecture delivers near-linear IOPS scaling with CPU count for NVMe workloads. The old sq (single-queue) path was completely removed in Linux 5.0.

/* Check how many hardware queues your NVMe has */
$ ls /sys/block/nvme0n1/mq/
0  1  2  3  4  5  6  7   # 8 hardware queues

$ cat /sys/block/nvme0n1/mq/0/cpu_list
0, 8, 16, 24             # CPUs mapped to HW queue 0

/* Show the scheduler for each device */
$ cat /sys/block/sda/queue/scheduler
mq-deadline [bfq] kyber none
#                ^^^^ active scheduler

$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber
#  ^^^^ none = no reordering, direct dispatch

/* Switch scheduler */
$ echo mq-deadline > /sys/block/sda/queue/scheduler

Writing a blk-mq driver is how you plug a storage device into the modern block layer. The driver must implement a small set of operations registered via struct blk_mq_ops.

The most important operation is .queue_rq(hctx, bd): this is called by the block layer when it wants the driver to dispatch a request to hardware. hctx is the hardware queue context, and bd->rq is the request to send. The driver should program the hardware (fill a submission queue ring, write to an MMIO register, etc.) and return BLK_STS_OK. If the hardware is busy (ring full), return BLK_STS_DEV_RESOURCE — blk-mq will retry.

Tag sets (struct blk_mq_tag_set) are the blk-mq mechanism for tracking in-flight requests. Each request gets a tag (an integer index, 0 to queue_depth-1). The tag lets the completion interrupt find the right struct request to complete. NVMe hardware uses exactly the same concept — the submission queue entry contains a command ID (= the tag) which is echoed back in the completion queue entry.

Key fields in blk_mq_tag_set:

• ops: pointer to your blk_mq_ops.
• nr_hw_queues: how many HW queues the device has.
• queue_depth: max in-flight requests per HW queue (NVMe default: 1023).
• cmd_size: extra bytes allocated per request for driver-private state (you can store your NVMe command or SCSI CDB there).

On completion, the driver calls blk_mq_end_request(rq, status) from the interrupt handler, which wakes waiters and releases the tag.

/* Skeleton of a minimal blk-mq driver (educational) */
#include <linux/blk-mq.h>
#include <linux/blkdev.h>

struct my_dev {
    struct blk_mq_tag_set  tag_set;
    struct request_queue  *queue;
    struct gendisk        *disk;
};

/* Called by blk-mq to dispatch a request to HW */
static blk_status_t my_queue_rq(struct blk_mq_hw_ctx *hctx,
                                 const struct blk_mq_queue_data *bd)
{
    struct request *rq = bd->rq;
    /* Tell block layer we started handling this request */
    blk_mq_start_request(rq);

    /* ... program DMA, write to HW submission ring, etc. ... */

    /* For simplicity: complete immediately (ramdisk case) */
    blk_mq_end_request(rq, BLK_STS_OK);
    return BLK_STS_OK;
}

static const struct blk_mq_ops my_mq_ops = {
    .queue_rq = my_queue_rq,
};

static int my_probe(struct device *dev)
{
    struct my_dev *mydev = /* ... */;

    mydev->tag_set.ops         = &my_mq_ops;
    mydev->tag_set.nr_hw_queues = 1;
    mydev->tag_set.queue_depth  = 128;
    mydev->tag_set.numa_node    = NUMA_NO_NODE;
    mydev->tag_set.cmd_size     = sizeof(struct my_hw_cmd);
    mydev->tag_set.flags        = BLK_MQ_F_SHOULD_MERGE;
    blk_mq_alloc_tag_set(&mydev->tag_set);

    mydev->queue = blk_mq_init_queue(&mydev->tag_set);
    /* ... add gendisk, set capacity, add_disk() ... */
    return 0;
}

An I/O scheduler sits between the software staging queues and the hardware dispatch queues. Its job: take a stream of incoming requests and decide which request to dispatch next, and when.

For spinning hard drives, this matters enormously. A 7200-RPM HDD's read head moves at ~50 mm/s and a random seek takes 5–15 ms. At 200 IOPS, spending 10 ms on each seek wastes nearly all throughput. The scheduler's job is to reorder requests so the head moves mostly in one direction (like an elevator servicing floors). This is why old schedulers were called elevator algorithms.

For SSDs and NVMe, there is no physical head. Any sector is reached in the same ~100 µs (SATA SSD) or ~10 µs (NVMe). Reordering does not help throughput. But schedulers still serve other purposes:

• Latency guarantees: prevent one process from starving another.
• Priority enforcement: interactive processes should get their I/Os served before background batch jobs.
• Write-ahead optimization: reorder to make writes hit the on-disk cache in a friendly pattern.

The current schedulers in blk-mq are: mq-deadline, BFQ, Kyber, and none (no scheduling, direct dispatch). Each has a radically different design philosophy.

/* See real I/O scheduling in action with blktrace */
/* This records what the block layer does for 5 seconds on /dev/sda */

$ sudo blktrace -d /dev/sda -o - | blkparse -i - 2>/dev/null | head -40

#  Output columns:
#  dev  cpu  seq  timestamp  pid  action  RWBS  sector+len  cmd
  8,0    2  1    0.000000000  1234  Q   R   2048 + 8    [dd]
  8,0    2  2    0.000003124  1234  G   R   2048 + 8    [dd]
  8,0    2  3    0.000003500  1234  I   R   2048 + 8    [dd]
  8,0    2  4    0.000003900  1234  D   R   2048 + 8    [dd]
  8,0    2  5    0.007123456  1234  C   R   2048 + 8   0
#
# Action codes:
#  Q = queued (bio submitted)
#  G = get request (bio -> request allocated)
#  I = inserted into scheduler queue
#  D = dispatched to driver
#  C = completed
#
# Q→G: bio allocated as request
# G→I: scheduler received it
# I→D: scheduler decided to dispatch it
# D→C: driver processed it (7ms here = typical HDD seek)

mq-deadline is the multi-queue successor to the classic deadline scheduler. Its core principle: every request must be dispatched within a bounded time — a deadline — regardless of what the scheduler would prefer to do for throughput. This prevents starvation.

The scheduler maintains two separate data structures:

1. A red-black tree sorted by sector number (for efficient sorted dispatch).
2. A FIFO queue sorted by submission time (for deadline enforcement).

Algorithm:

1. If any request has passed its deadline, dispatch the oldest expired request immediately.
2. Otherwise, dispatch the next request in sector order (the "next" pointer advances like an elevator head sweeping one direction).

Deadlines: reads get 500 ms by default; writes get 5000 ms. Reads are prioritized over writes by default (read_expire < write_expire) because a process waiting on a read() is typically blocking, while writes are often async.

Write batching: to avoid the head oscillating between read and write sectors, mq-deadline processes writes in batches (writes_starved: process up to fifo_batch writes before switching back to reads). This is tunable via /sys/block/sda/queue/iosched/writes_starved.

When to use it: spinning HDDs and SATA SSDs where you need guaranteed latency bounds. It is the correct choice for databases (PostgreSQL, MySQL fsync-heavy workloads) where a write that misses its deadline causes a timeout cascade. Also good for real-time applications. NOT ideal for desktops with mixed interactive + background workloads (BFQ is better there).

/* Tune mq-deadline on an HDD for database workload */

# Check current deadline settings
$ cat /sys/block/sda/queue/iosched/read_expire
500    # milliseconds
$ cat /sys/block/sda/queue/iosched/write_expire
5000   # milliseconds
$ cat /sys/block/sda/queue/iosched/writes_starved
2      # process max 2 write batches before reads get priority
$ cat /sys/block/sda/queue/iosched/fifo_batch
16     # max requests per batch dispatch

# For a database under heavy concurrent reads:
# Tighten the read deadline, loosen writes
$ echo 250  > /sys/block/sda/queue/iosched/read_expire
$ echo 2000 > /sys/block/sda/queue/iosched/write_expire

# For a write-heavy log disk: reverse the priorities
$ echo 500  > /sys/block/sda/queue/iosched/write_expire
$ echo 5000 > /sys/block/sda/queue/iosched/read_expire
$ echo 1    > /sys/block/sda/queue/iosched/writes_starved

BFQ (Budget Fair Queuing) is the most sophisticated scheduler in the Linux kernel. It was designed by Paolo Valente (University of Modena) and merged in Linux 4.12. Its goal is proportional-share scheduling of I/O bandwidth, with special attention to low latency for interactive applications.

The fundamental analogy: BFQ thinks of processes as queues competing for disk time, and assigns each queue a budget (measured in sectors). The scheduler lets a queue exhaust its budget before switching to the next — similar to how a CPU scheduler runs a process for a full time quantum. This batching reduces seek distance on HDDs (a key win over round-robin per-request schemes).

BFQ's key features:

1. Weight-based shares: each process gets a weight (default 100, configurable via ionice / cgroup I/O weight). A process with weight 200 gets twice the bandwidth of one with weight 100. This integrates with Linux's I/O cgroup (io.weight).

2. Idle detection and low-latency heuristics: if a process has been I/O-active but suddenly goes idle (e.g., it's waiting for the disk), BFQ waits a short time (slice_idle, default 8 ms on HDDs) before switching to the next queue. This avoids "injecting seeks" from other processes while the current one is about to submit more requests. It is the critical mechanism that makes interactive desktops smooth under heavy background I/O.

3. Anti-starvation: despite idle waiting, BFQ tracks budget expiry and prevents starvation.

4. Hierarchical scheduling: integrates with Linux cgroups. You can give a cgroup 80% of disk bandwidth and another 20%, and BFQ enforces this proportionally.

When to use it: desktop Linux where you want mplayer to not stutter when make -j16 is burning the disk in the background. Also correct for Android (BFQ is the default there). Not the best choice for a pure database server (mq-deadline's hard deadlines are more predictable there) or for NVMe (overhead exceeds benefit).

/* Demonstrate BFQ I/O priority via ionice */

# ionice maps to I/O scheduling class:
# Class 1 = real-time, Class 2 = best-effort, Class 3 = idle

# Run a background sync at idle priority (won't interfere with UI)
$ ionice -c 3 rsync -av /source /dest &

# Run a foreground database at real-time I/O priority
$ ionice -c 1 -n 0 ./my_database

# Check which class a PID is using
$ ionice -p $(pgrep rsync)
idle: prio 0

# With BFQ + cgroups v2: set I/O weight per-cgroup
# (Must be using BFQ scheduler)
$ echo '8:0 100' > /sys/fs/cgroup/mygroup/io.weight  # 10% of weight 1000
$ echo '8:0 800' > /sys/fs/cgroup/dbgroup/io.weight  # 80% of weight 1000

# BFQ tuning
$ cat /sys/block/sda/queue/iosched/slice_idle
8     # ms to wait for an idle process (reduces seeks)
$ cat /sys/block/sda/queue/iosched/low_latency
1     # enable low-latency heuristics (default on)

Kyber (merged in Linux 4.12 alongside BFQ) takes a completely different philosophical stance: instead of understanding workloads and fairness deeply, it trusts the hardware and uses token-bucket feedback control to maintain target latency percentiles.

Kyber maintains two token buckets: one for reads and one for writes. Tokens represent permission to dispatch a request. The bucket sizes (read_lat_nsec and write_lat_nsec, default 2 ms and 10 ms respectively) encode the target latency. Each time a request completes, Kyber measures its latency:

• If latency was good (under target): refill tokens aggressively → allow high concurrency.
• If latency was bad (over target): drain tokens → throttle dispatch → give the hardware breathing room.

This is a feedback control loop — like a proportional controller (P-controller) in control theory. The system observes an output (latency), compares to a setpoint (target), and adjusts the input (dispatch rate).

Kyber also separates reads from writes into two independent priority classes and dispatches them round-robin, preventing writes from drowning reads.

When to use it: Kyber is designed for fast, low-latency devices — NVMe SSDs, high-end SATA SSDs — where the hardware can handle massive concurrency and the scheduler's job is just to avoid overwhelming it. On NVMe, Kyber adds very little overhead compared to BFQ's complex per-process tracking, making it better for latency-sensitive server workloads (databases, real-time services). It is the right scheduler when you want predictable P99 latency on NVMe.

The none scheduler: for highest-throughput NVMe workloads where the application handles its own I/O ordering (databases using O_DIRECT, io_uring with fixed buffers), none (direct dispatch, no scheduler) gives maximum throughput with minimum overhead.

/* Kyber latency targets (in nanoseconds) */
$ cat /sys/block/nvme0n1/queue/iosched/read_lat_nsec
2000000    # 2 ms target for reads

$ cat /sys/block/nvme0n1/queue/iosched/write_lat_nsec
10000000   # 10 ms target for writes

# Tune for a more aggressive latency target (fast NVMe, DB workload)
$ echo 1000000 > /sys/block/nvme0n1/queue/iosched/read_lat_nsec

# Compare schedulers: measure P99 latency with fio
$ fio --name=read_lat_test --rw=randread --bs=4k \
      --iodepth=32 --numjobs=4 \
      --filename=/dev/nvme0n1 --direct=1 \
      --runtime=30 --time_based \
      --output-format=json \
      --output=kyber_results.json
# Then check: .jobs[0].read.clat_ns.percentile["99.000000"]
# Repeat after: echo bfq > /sys/block/nvme0n1/queue/scheduler
# Compare the P99 values.

Now let's trace a single read(fd, buf, 4096) syscall all the way to the hardware and back. This is the complete integration of everything above.

Step 1 — VFS layer: read() enters the kernel via sys_read() → vfs_read() → generic_file_read_iter(). The VFS checks the page cache first. If the page is already there (cache hit), the data is copied to userspace and we're done — no block I/O happens. This is why cat file; cat file is fast.

Step 2 — Page cache miss: if the page is not cached, generic_file_read_iter() calls ext4_readpage() (or the appropriate filesystem's readpage). The filesystem maps the file's logical block number to a physical sector on the block device (using the inode's extent tree or block map). It then calls submit_bio().

Step 3 — bio construction: submit_bio() calls generic_make_request() (now submit_bio_noacct()), which enters blk_mq_make_request(). This function allocates a struct request from the tag set's per-CPU pool, copies the bio's fields, and attempts a merge with existing requests in the per-CPU software staging queue.

Step 4 — Scheduler insertion: if no merge is possible, the request is inserted into the I/O scheduler's data structures (BFQ's per-process tree, mq-deadline's red-black tree, etc.) via elv_merge() and elv_add_request().

Step 5 — Dispatch: the scheduler decides to dispatch (either because the request's deadline has expired, its budget is used, or a batching opportunity exists). It calls the driver's .queue_rq() operation.

Step 6 — Driver / hardware: the NVMe driver fills a submission queue entry (SQE) with the command, physical addresses (from the bio's scatter-gather list), and the request's tag. It rings the doorbell register (a single MMIO write). The NVMe controller DMAs the data from disk to the page cache page.

Step 7 — Completion: the NVMe controller writes a completion queue entry (CQE) with the tag and status. The interrupt handler fires, calls blk_mq_end_request(), which invokes the bio's bi_end_io callback, which wakes the process sleeping in generic_file_read_iter(). The page is now in the page cache.

Step 8 — Copy to userspace: the woken process copies from the (now populated) page cache page into the user buffer. read() returns.

/* Trace the I/O path with ftrace (function graph tracer) */
/* This shows the actual kernel call chain for a read() */

$ cd /sys/kernel/debug/tracing
$ echo function_graph > current_tracer
$ echo 'submit_bio blk_mq_make_request blk_mq_dispatch_rq_list'\
       > set_graph_function
$ echo 1 > tracing_on
$ cat /tmp/testfile > /dev/null  # trigger I/O
$ echo 0 > tracing_on
$ head -50 trace

# Typical output (abbreviated):
# submit_bio() {
#   blk_mq_make_request() {
#     blk_mq_get_request();
#     blk_mq_bio_to_request();
#     elv_attempt_insert_merge() {
#       bfq_merge();
#     }
#     blk_mq_sched_insert_request() {
#       bfq_insert_requests();
#     }
#     blk_mq_run_hw_queue() {
#       blk_mq_dispatch_rq_list() {
#         nvme_queue_rq();  /* driver dispatch */
#       }
#     }
#   }
# }

One subtle but important optimization in the block layer is plugging. When a process is about to submit multiple I/Os (e.g., a filesystem reading several blocks for a large file), it is wasteful to dispatch each bio immediately — the scheduler cannot merge them if they have already been dispatched.

Plugging lets the caller defer dispatch: blk_start_plug(struct blk_plug *plug) tells the block layer "I am about to submit a batch; hold the requests in this per-task plug list instead of dispatching them immediately." Then blk_finish_plug(plug) flushes the plug — tries final merges and dispatches everything at once.

This is analogous to how writev() is faster than multiple write() calls: you batch the work, then do one expensive operation.

Kernel code for filesystem readahead does exactly this:

blk_start_plug(&plug);
for each page to prefetch:
    submit_bio(bio for that page);
blk_finish_plug(&plug);  /* dispatch all at once */

The unplug also happens automatically: if the task goes to sleep (e.g., waits for I/O), the block layer automatically calls blk_finish_plug() before sleeping — the scheduler notices via task_struct->plug.

This mechanism interacts beautifully with blk-mq's per-CPU queues: plugging accumulates requests in the per-CPU software queue; unplugging runs the hardware queue, potentially merging many requests into a single large DMA transfer. On a sequential read of a 1 GB file, this is the difference between issuing 256K individual 4KB requests and issuing a handful of large 512KB requests — easily 10× throughput difference on HDDs.

/* Plugging in kernel code (from mm/readahead.c pattern) */
#include <linux/blkdev.h>

void prefetch_pages(struct address_space *mapping,
                    pgoff_t start, unsigned int nr_pages)
{
    struct blk_plug plug;
    unsigned int i;

    /* Tell block layer: hold requests, don't dispatch yet */
    blk_start_plug(&plug);

    for (i = 0; i < nr_pages; i++) {
        struct page *page = /* allocate/find cache page */;
        /* submit_bio internally — goes into plug list */
        mapping->a_ops->readpage(NULL, page);
    }

    /* Now: merge everything, dispatch in one shot */
    blk_finish_plug(&plug);
    /* Hardware sees: one large sequential request,
       not hundreds of small ones. */
}