io_uring: Modern Async I/O

To appreciate io_uring, you must feel the pain it solved. Linux had several async I/O mechanisms before it, and each was deeply flawed.

read()/write() — synchronous, blocking When you call read(fd, buf, n), the calling thread blocks until data arrives. The kernel puts your thread to sleep in a wait queue. This is fine for a simple program, but a server handling 10,000 concurrent connections cannot afford 10,000 sleeping threads — each thread needs at least 8 KB of kernel stack and tens of kilobytes of userspace stack.

select()/poll()/epoll() — readiness notification, not completion These interfaces tell you "this file descriptor is ready to read" — but they don't actually do the I/O. You still need a separate read() call after epoll_wait() returns. That is two round trips to the kernel per I/O operation. For a server doing 1 million operations per second, that is 2 million syscalls per second. Each syscall costs ~100–300 ns in context switches and TLB/cache effects.

POSIX AIO (aio_read, aio_write) — broken in practice POSIX AIO looks async on the surface: submit a request, get notified via signal or polling when it completes. But the Linux implementation used a pool of userspace threads that each did blocking I/O — it was fake async. It also only worked for O_DIRECT file I/O, not sockets, not pipes, not regular buffered files. Nobody used it for serious work.

Linux native AIO (io_submit/io_getevents) — limited and slow Linux's own native AIO syscalls worked, but only for O_DIRECT (bypassing the page cache), only for files (not sockets), required memaligned buffers, had per-context limits, and still required at least one syscall per batch of submissions and one per batch of completions. Databases like PostgreSQL and MySQL tried it, found it painful, and kept falling back to thread pools.

The common theme: every operation needs at least one syscall, every notification system is separate from the submission system, and most interfaces cover only a subset of I/O types. io_uring attacked all of these problems simultaneously.

/* Old-style epoll + read: two kernel round trips per I/O */
int fd = open("file.txt", O_RDONLY | O_NONBLOCK);

/* Round trip 1: register interest */
struct epoll_event ev = { .events = EPOLLIN, .data.fd = fd };
epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev);  /* syscall 1 */

/* Round trip 2: wait for readiness */
struct epoll_event events[10];
int n = epoll_wait(epfd, events, 10, -1); /* syscall 2 */

/* Round trip 3: actually read */
char buf[4096];
read(fd, buf, sizeof(buf));               /* syscall 3 */
/* Total: 3 syscalls for one read. io_uring does this with 0 or 1. */

io_uring's core innovation is two lock-free, single-producer/single-consumer ring buffers shared between the kernel and userspace via mmap. Let us be precise about what this means.

Submission Queue (SQ) The application is the producer of the SQ. The kernel is the consumer. When you want to do I/O, you write a Submission Queue Entry (SQE) into the ring and advance the tail pointer. The kernel reads from the head and advances it as it processes entries. No lock is needed because each side only writes to one pointer (tail or head) and reads the other.

Completion Queue (CQ) The kernel is the producer of the CQ. The application is the consumer. When the kernel finishes an operation, it writes a Completion Queue Entry (CQE) and advances the CQ tail. The application reads CQEs from the head and advances it. Again, no lock.

The SQ Indirection Array There is a subtlety: the SQ ring does not contain SQEs directly. Instead, it contains indices into a separate SQE array. This is important because it lets the kernel process SQEs out of order (for maximum throughput) while the application can still pre-fill SQEs in any array slot it likes. The ring is just the ordering queue; the SQEs themselves live in a flat array.

Memory layout After io_uring_setup(), you mmap two regions from the kernel:

1. IORING_OFF_SQ_RING — contains the SQ ring header (head, tail, ring_mask, flags, array) and the CQ ring (head, tail, ring_mask, cqes[]).
2. IORING_OFF_SQES — the flat SQE array.

Both regions are real kernel memory mapped read-write into your process. The kernel and your process literally share the same physical pages. No copying ever happens.

Why this enables zero-syscall I/O Because both sides can write to and read from these shared pages directly, the application can submit hundreds of SQEs and read hundreds of CQEs without a single syscall — as long as the kernel is polling these rings in a kernel thread (IORING_SETUP_SQPOLL). Without SQPOLL, you need one io_uring_enter() call to tell the kernel to start processing, but you can batch arbitrarily many SQEs into that one call.

/* Conceptual layout after io_uring_setup() + mmap */

/* Shared SQ ring (in mapped kernel memory) */
struct io_sq_ring {
    uint32_t head;        /* kernel advances this (consumer) */
    uint32_t tail;        /* app advances this (producer) */
    uint32_t ring_mask;   /* = entries - 1, for wrapping */
    uint32_t ring_entries;
    uint32_t flags;
    uint32_t dropped;
    uint32_t array[];     /* indices into SQE array */
};

/* Flat SQE array (separate mmap region) */
struct io_uring_sqe sqes[entries]; /* app writes here */

/* Shared CQ ring */
struct io_cq_ring {
    uint32_t head;        /* app advances this (consumer) */
    uint32_t tail;        /* kernel advances this (producer) */
    uint32_t ring_mask;
    uint32_t ring_entries;
    uint32_t overflow;
    struct io_uring_cqe cqes[]; /* kernel writes here */
};

/* To submit: write SQE, update sq_tail (with memory barrier) */
/* To collect: read CQE, update cq_head (with memory barrier) */

Let us look at what actually goes in these entries, because the design is clever and tells you a lot about what io_uring can do.

SQE (Submission Queue Entry) — 64 bytes Every I/O request you want to make is encoded in one SQE. The opcode field selects the operation (IORING_OP_READ, IORING_OP_WRITE, IORING_OP_ACCEPT, IORING_OP_CONNECT, IORING_OP_FSYNC, IORING_OP_TIMEOUT, and dozens more). Other fields supply the arguments: fd, buffer address (addr), length (len), file offset (off), flags, and a 64-bit user_data tag that you choose — it comes back in the CQE so you know which request completed.

The union fields handle the variety of operations: addr can be a buffer address (for read/write), a sockaddr pointer (for connect), or a string pointer (for openat). off is a file offset for files or 0 for sockets. flags can set IOSQE_FIXED_FILE (use a registered file descriptor), IOSQE_IO_LINK (chain this to the next SQE), IOSQE_ASYNC (force async even if the operation could complete inline), and more.

CQE (Completion Queue Entry) — 16 bytes Much simpler: user_data (the tag you put in the SQE — this is how you match completions to requests), res (the return value, exactly like what the equivalent syscall would return — negative errno on error, bytes transferred on success), and flags (rarely used, but can carry extra info for some ops).

The asymmetry in size (64 vs 16 bytes) makes sense: submissions need all the arguments; completions just need the result and an identifier.

The user_data field is your handle Because the kernel processes SQEs potentially out of order, you cannot assume CQE[0] corresponds to SQE[0]. The user_data field is how you correlate them. A common pattern is to store a pointer to a context struct in user_data, cast it back in the completion handler. This is safe because user_data is 64 bits, which is pointer-sized on 64-bit systems.

/* From <linux/io_uring.h> — the actual kernel struct */
struct io_uring_sqe {
    __u8    opcode;     /* IORING_OP_READ, _WRITE, _ACCEPT, etc. */
    __u8    flags;      /* IOSQE_FIXED_FILE, IOSQE_IO_LINK, ... */
    __u16   ioprio;
    __s32   fd;         /* file descriptor (or registered fd index) */
    union { __u64 off; __u64 addr2; }; /* file offset */
    union { __u64 addr; __u64 splice_off_in; }; /* buffer or sockaddr */
    __u32   len;        /* buffer length */
    /* opcode-specific flags in a union... */
    __u64   user_data;  /* YOU choose; comes back in CQE */
    /* 16 bytes of opcode-specific extra fields */
};

struct io_uring_cqe {
    __u64   user_data;  /* copied from SQE — your correlation key */
    __s32   res;        /* like syscall return: bytes or -errno */
    __u32   flags;      /* IORING_CQE_F_MORE, etc. */
};

/* Example: set up a read SQE with a context pointer as user_data */
struct my_request { char buf[4096]; int fd; };
struct my_request *req = malloc(sizeof(*req));
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, req->fd, req->buf, sizeof(req->buf), 0);
sqe->user_data = (uint64_t)(uintptr_t)req; /* store pointer! */

The raw io_uring interface (just io_uring_setup, io_uring_enter, and io_uring_register syscalls plus manual mmaps) is powerful but verbose. Jens Axboe also wrote liburing, a thin wrapper library that handles the mmap bookkeeping, provides helper functions for preparing each operation type, and adds convenience wrappers. This is what you should use for application code. It does not hide the rings — it just manages them for you.

The lifecycle

1. io_uring_queue_init(entries, &ring, flags) — calls io_uring_setup, does the mmaps, sets up bookkeeping.
2. io_uring_get_sqe(&ring) — returns a pointer to the next available SQE slot.
3. io_uring_prep_*(sqe, ...) — fills in the SQE fields for the specific operation.
4. io_uring_submit(&ring) — calls io_uring_enter to notify the kernel (may be skipped with SQPOLL).
5. io_uring_wait_cqe(&ring, &cqe) — blocks until a CQE is available, fills cqe.
6. io_uring_cqe_seen(&ring, cqe) — advances the CQ head (marks CQE as consumed).
7. io_uring_queue_exit(&ring) — cleans up.

Batching The real power comes from batching: call io_uring_get_sqe and io_uring_prep_* multiple times before calling io_uring_submit once. All those operations go to the kernel in one syscall. Similarly, use io_uring_peek_cqe in a loop to drain all available completions without additional syscalls.

Kernel version sensitivity Different io_uring features require different kernel versions. IORING_OP_READ/WRITE: 5.6. Multishot accept: 5.19. Fixed wait: 5.11. Always check io_uring_opcode_supported() or probe with io_uring_probe before using newer ops.

#include <liburing.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(void) {
    struct io_uring ring;
    /* Initialize ring with 8 SQE slots */
    io_uring_queue_init(8, &ring, 0);

    int fd = open("/etc/hostname", O_RDONLY);
    char buf[256] = {0};

    /* Get an SQE slot and fill it for a read */
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fd, buf, sizeof(buf) - 1, 0);
    sqe->user_data = 42; /* arbitrary tag */

    /* Submit: one syscall for potentially many SQEs */
    io_uring_submit(&ring);

    /* Wait for completion */
    struct io_uring_cqe *cqe;
    io_uring_wait_cqe(&ring, &cqe);

    if (cqe->res < 0)
        fprintf(stderr, "Error: %s\n", strerror(-cqe->res));
    else
        printf("Read %d bytes: %s", cqe->res, buf);

    io_uring_cqe_seen(&ring, cqe); /* advance CQ head */
    io_uring_queue_exit(&ring);
    return 0;
}
/* Compile: gcc -o hello_uring hello_uring.c -luring */

The headline feature of io_uring is the ability to do I/O with zero syscalls after setup. This requires two things working together: SQPOLL mode and registered (fixed) buffers/files.

SQPOLL mode (IORING_SETUP_SQPOLL) When you pass IORING_SETUP_SQPOLL to io_uring_setup, the kernel spawns a dedicated kernel thread (an io_uring-sq kernel thread, visible in ps) that continuously polls the SQ ring. When your application writes a new SQE and advances the SQ tail, the kernel thread sees it — no io_uring_enter call needed. The kernel thread spins for a configurable sq_thread_idle period (default 1 second), then sleeps until you call io_uring_enter to wake it. This means: for a busy server, zero syscalls for submissions. For the completion side, you just poll the CQ ring yourself — also zero syscalls.

The tradeoff: a spinning kernel thread consumes a CPU core. SQPOLL makes sense for applications that are doing I/O constantly (network servers, databases, storage engines) where trading a CPU core for eliminated syscall overhead is a win. For occasional I/O, the normal mode is better.

Registered Buffers (io_uring_register_buffers) Every time you submit a read or write with a regular userspace buffer, the kernel must do two expensive things: validate the userspace pointer and then pin the physical pages in memory (so they cannot be swapped out while DMA is in progress). For millions of operations per second, this per-operation page-pinning is a real cost.

With io_uring_register_buffers, you pre-register a set of buffers with the kernel once. The kernel pins and maps those pages at registration time and keeps them pinned for the lifetime of the ring. Subsequent reads/writes using IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXED skip the pin/unpin step entirely — the kernel already knows those pages are safe.

Registered Files Similarly, io_uring_register_files pre-registers a set of file descriptors. When using a registered fd (via IOSQE_FIXED_FILE in the SQE flags), the kernel skips the fdget/fdput reference counting on every operation — another small but cumulative saving at high IOPS.

The Full Zero-Syscall Path With SQPOLL + fixed buffers + fixed files: after io_uring_setup and the registration calls, your application can run an entire I/O loop (write SQEs, read CQEs) without a single syscall. The shared memory rings are the only communication channel. This is as close as Linux gets to kernel-bypass I/O without DPDK or SPDK — but it works with the full kernel network/storage stack, not bypassing it.

/* SQPOLL + fixed buffer setup sketch */
#include <liburing.h>

#define BUFSIZE (4096)
#define NBUFS   (64)

char fixed_bufs[NBUFS][BUFSIZE];

struct io_uring ring;
struct io_uring_params params = {
    .flags = IORING_SETUP_SQPOLL,
    .sq_thread_idle = 2000, /* kernel thread idles 2s before sleeping */
};
io_uring_queue_init_params(128, &ring, &params);

/* Register buffers: kernel pins these pages once */
struct iovec iovecs[NBUFS];
for (int i = 0; i < NBUFS; i++) {
    iovecs[i].iov_base = fixed_bufs[i];
    iovecs[i].iov_len  = BUFSIZE;
}
io_uring_register_buffers(&ring, iovecs, NBUFS);

/* Later: submit a fixed-buffer read — NO io_uring_enter needed! */
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
/* Use READ_FIXED: buf_index 3 = fixed_bufs[3] */
io_uring_prep_read_fixed(sqe, fd, fixed_bufs[3], BUFSIZE, 0, 3);
/* Just advance the tail — SQPOLL kernel thread sees it */
io_uring_submit(&ring); /* can be skipped if SQPOLL is spinning */

/* Poll CQ without a syscall */
struct io_uring_cqe *cqe;
while (io_uring_peek_cqe(&ring, &cqe) == -EAGAIN)
    ; /* busy-poll — no syscall */
printf("result: %d\n", cqe->res);
io_uring_cqe_seen(&ring, cqe);

One of io_uring's most elegant features is the ability to chain operations so that the next operation only starts if the previous one succeeded. This lets you express complex I/O patterns — open a file, then read from it, then close it — as a single atomic chain submitted in one io_uring_submit call, without your application needing to wake up between steps.

Hard Links (IOSQE_IO_LINK) Set IOSQE_IO_LINK in an SQE's flags to link it to the next SQE in the submission. If the first operation succeeds, the second proceeds. If the first fails (non-zero error result), all subsequent linked operations are cancelled (they complete with -ECANCELED). The chain continues until you reach an SQE without IOSQE_IO_LINK.

Hard Drain (IOSQE_IO_DRAIN) This flag tells io_uring to not start this SQE until ALL previously submitted SQEs have completed. It is a full barrier, ensuring ordering for correctness-sensitive operations (like: finish all writes before issuing an fsync).

Practical example: atomic open-read-close You can submit three linked SQEs — openat, read, close — and the kernel will execute them in sequence, short-circuiting if any step fails. This replaces three separate async rounds in old-style epoll code with a single submission that completes as one logical unit.

Timeout cancellation with links A powerful pattern: link a normal operation SQE to a IORING_OP_LINK_TIMEOUT SQE. If the main operation completes before the timeout, the timeout is cancelled. If the timeout fires first, the main operation is cancelled. This gives you per-operation timeouts without a separate timeout management thread — just one chained pair of SQEs.

/* Linked open -> read -> close: all in one submit call */
#include <liburing.h>
#include <fcntl.h>

void linked_open_read_close(struct io_uring *ring,
                             const char *path,
                             char *buf, size_t len)
{
    /* SQE 1: openat */
    struct io_uring_sqe *sqe1 = io_uring_get_sqe(ring);
    io_uring_prep_openat(sqe1, AT_FDCWD, path, O_RDONLY, 0);
    sqe1->flags |= IOSQE_IO_LINK; /* link to next */
    sqe1->user_data = 1;

    /* SQE 2: read from the fd returned by openat */
    /* Note: fd=-1 here; the kernel fills it from sqe1's result */
    struct io_uring_sqe *sqe2 = io_uring_get_sqe(ring);
    io_uring_prep_read(sqe2, -1, buf, len, 0);
    sqe2->flags |= IOSQE_IO_LINK | IOSQE_FIXED_FILE;
    /* IOSQE_CQE_SKIP_SUCCESS to not get a CQE for the open */
    sqe2->flags |= IOSQE_CQE_SKIP_SUCCESS; /* kernel 5.17+ */
    sqe2->user_data = 2;

    /* SQE 3: close — only runs if read succeeded */
    struct io_uring_sqe *sqe3 = io_uring_get_sqe(ring);
    io_uring_prep_close_direct(sqe3, 0);
    sqe3->user_data = 3;
    /* No IOSQE_IO_LINK: end of chain */

    io_uring_submit(ring);
    /* Now wait for completions... */
}

Classic io_uring: one SQE → one CQE. You read once, get one completion, submit the next read. For a server accepting connections, this means: accept one connection, get CQE, submit another accept SQE, get CQE... This is fine but still involves re-submitting after each event.

Multishot operations change the model: one SQE → many CQEs. The SQE stays active in the kernel until you cancel it or an error occurs. Each time the underlying event happens, the kernel posts a new CQE. This is ideal for:

• IORING_OP_ACCEPT_MULTISHOT: one accept SQE that keeps accepting new connections forever. Each accepted connection gets a CQE with the new fd. The IORING_CQE_F_MORE flag in the CQE's flags field tells you the SQE is still active.
• IORING_OP_RECV_MULTISHOT: one recv SQE that posts a CQE for each incoming packet/message.
• IORING_OP_POLL_MULTISHOT: poll a fd for events indefinitely.

Buffer rings for multishot A challenge: for multishot recv, the kernel needs a buffer ready for each incoming message, but you don't know when messages will arrive. The IORING_OP_PROVIDE_BUFFERS operation (and the newer buffer ring API) lets you pre-register a pool of buffers with the kernel. When a multishot recv fires, the kernel picks a buffer from the pool, fills it, and tells you which buffer it used via the CQE flags (IORING_CQE_F_BUFFER and the buffer ID in the upper 16 bits of flags). This avoids any buffer allocation on the hot path.

The combination of multishot accept + multishot recv + buffer rings is what modern io_uring-based HTTP servers use. The application's hot loop is pure CQE consumption — no submitting new SQEs, no buffer management, just reading completions and processing data.

/* Multishot accept: one SQE accepts forever */
#include <liburing.h>
#include <sys/socket.h>

void setup_multishot_accept(struct io_uring *ring, int server_fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    /* addr=NULL: don't capture client address (for simplicity) */
    io_uring_prep_multishot_accept(sqe, server_fd, NULL, NULL, 0);
    sqe->user_data = 100; /* tag for accept completions */
    io_uring_submit(ring);
}

void event_loop(struct io_uring *ring) {
    struct io_uring_cqe *cqe;
    while (1) {
        io_uring_wait_cqe(ring, &cqe);
        if (cqe->user_data == 100) {
            int client_fd = cqe->res; /* new connection */
            int more = cqe->flags & IORING_CQE_F_MORE;
            /* handle new connection... */
            if (!more) {
                /* multishot ended (error) — re-arm */
                setup_multishot_accept(ring, server_fd);
            }
            /* No need to re-submit accept SQE! */
        }
        io_uring_cqe_seen(ring, cqe);
    }
}

liburing is great for application code, but kernel contributors need to understand the raw syscalls. There are exactly three:

io_uring_setup(entries, params) → fd Creates a new io_uring instance with entries SQ slots (rounded up to the next power of two). params is a struct io_uring_params where you pass setup flags (IORING_SETUP_SQPOLL, IORING_SETUP_IOPOLL, IORING_SETUP_SINGLE_ISSUER, etc.) and receive back the actual ring sizes and feature flags the kernel supports. Returns a file descriptor representing this io_uring instance — you can pass it over Unix sockets, inherit it across fork, etc.

io_uring_enter(fd, to_submit, min_complete, flags, sig) → int The main workhorse syscall. to_submit says how many SQEs to process from the SQ. min_complete says how many CQEs to wait for before returning. flags can include IORING_ENTER_GETEVENTS (block waiting for completions). This single syscall can simultaneously drain the SQ and wait for completions — in the common case, you call this once per batch, not once per operation.

io_uring_register(fd, opcode, arg, nr_args) → int Registers/unregisters resources: buffers (IORING_REGISTER_BUFFERS), files (IORING_REGISTER_FILES), event fds, and more. Registration is done once and the kernel holds references to those resources.

The io_uring_params feature flags The features field returned by io_uring_setup is a bitmask telling you what the running kernel supports: IORING_FEAT_SQPOLL_NONFIXED (SQPOLL without needing root), IORING_FEAT_NATIVE_WORKERS (kernel uses native workers instead of a thread pool), IORING_FEAT_NODROP (CQE overflow goes to an internal overflow list rather than dropping), etc. Always check these before using advanced features — a binary might run on kernels ranging from 5.1 to 6.x.

/* Raw syscall interface — what liburing wraps */
#include <sys/syscall.h>
#include <linux/io_uring.h>
#include <sys/mman.h>

/* SYS_io_uring_setup = 425, SYS_io_uring_enter = 426,
   SYS_io_uring_register = 427  (on x86-64) */

int ring_fd;
struct io_uring_params params = {};
/* Request SQPOLL; kernel fills in sq_off, cq_off, sq_entries, etc. */
params.flags = IORING_SETUP_SQPOLL;
params.sq_thread_idle = 1000; /* ms */

ring_fd = syscall(SYS_io_uring_setup, 128, &params);
/* params.features now has IORING_FEAT_* bitmask */

/* Map the SQ ring and CQ ring (may be in same mapping on newer kernels) */
void *sq_ring = mmap(NULL,
    params.sq_off.array + params.sq_entries * sizeof(uint32_t),
    PROT_READ | PROT_WRITE,
    MAP_SHARED | MAP_POPULATE,
    ring_fd, IORING_OFF_SQ_RING);

void *sqes = mmap(NULL,
    params.sq_entries * sizeof(struct io_uring_sqe),
    PROT_READ | PROT_WRITE,
    MAP_SHARED | MAP_POPULATE,
    ring_fd, IORING_OFF_SQES);

/* Enter: submit 1 SQE, wait for 1 completion */
syscall(SYS_io_uring_enter, ring_fd, 1, 1,
        IORING_ENTER_GETEVENTS, NULL, 0);

io_uring has had a troubled security history, and understanding why is important both for kernel contributors and for anyone deploying it in production.

The core tension io_uring gives userspace the ability to ask the kernel to perform arbitrary operations on its behalf — operations that run in kernel context, with kernel credentials, asynchronously. This is a powerful capability, and it creates a large attack surface.

Credential passing and privilege escalation The most serious class of bugs: io_uring operations run asynchronously, sometimes in the context of a different thread or a kernel worker. Early implementations sometimes ran operations with the wrong credentials — the credentials at submission time vs. the credentials at execution time. CVE-2023-2598 and related issues exploited this to perform privilege escalation: a low-privilege process could submit an operation that would run with elevated privileges because of a credential confusion bug.

Use-After-Free via async operations Another class: an SQE references userspace memory (buffer addresses, sockaddrs, etc.). If the application frees that memory before the async operation completes, the kernel reads freed memory. io_uring mitigates this with IORING_OP_* operations that copy the argument (e.g., for connect, the sockaddr is copied into kernel memory at submission time). But bugs in this copy-on-submission code have led to UAF vulnerabilities.

Security response: restrictions Because of these issues:

• Android disabled io_uring for most processes (only available to privileged processes).
• ChromeOS disabled it in its seccomp sandbox.
• The kernel added io_uring_disabled sysctl (and io_uring_group restriction) so administrators can restrict io_uring to privileged processes.
• The IORING_SETUP_SINGLE_ISSUER flag was added (5.20/6.0) to tighten the model: only the thread that created the ring may submit to it.
• Security-conscious runtime environments (container runtimes, sandboxes) often block io_uring_setup via seccomp.

What this means for kernel contributors Every new io_uring operation must be carefully reviewed for: (1) credential handling — does it capture and restore credentials correctly? (2) argument lifetime — are userspace pointers dereferenced only at submission time or later? (3) cancellation — if the operation is cancelled, are all kernel resources released? The io_uring subsystem has a high bar for new operations precisely because past mistakes were costly.

This does not mean io_uring is unsafe — the known CVEs have been fixed, and the architecture has been hardened. But it is a reminder that powerful kernel interfaces deserve careful security thinking.

/* Check kernel's io_uring security knobs */
/* /proc/sys/kernel/io_uring_disabled:
   0 = enabled for all (default)
   1 = disabled for non-CAP_SYS_ADMIN
   2 = disabled for all

   Also: /proc/sys/kernel/io_uring_group (gid allowed)
*/

/* In a seccomp filter (e.g., for a container runtime):
   Blocking io_uring_setup prevents any use of io_uring:
*/
/* In C with libseccomp: */
/* seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM),
                   SCMP_SYS(io_uring_setup), 0); */

/* Check if your kernel supports io_uring and probe capabilities: */
#include <liburing.h>
struct io_uring_probe *probe = io_uring_get_probe();
if (probe) {
    printf("IORING_OP_READ supported: %d\n",
           io_uring_opcode_supported(probe, IORING_OP_READ));
    printf("IORING_OP_ACCEPT_MULTISHOT supported: %d\n",
           io_uring_opcode_supported(probe,
               IORING_OP_ACCEPT));
    io_uring_free_probe(probe);
}

Understanding how the kernel implements io_uring helps you read patches and reason about performance. Let us trace what happens when io_uring_enter is called.

Syscall entry: io_uring_enter In the kernel, io_uring_enter lands in io_uring_enter() in io_uring/io_uring.c. It finds the io_ring_ctx structure associated with the ring fd. This structure holds pointers to both rings, the registered buffers/files tables, the SQPOLL thread state, and the completion list.

SQ processing: io_submit_sqes The kernel iterates over the SQ ring, reading index entries to find SQEs. For each SQE, it calls io_submit_sqe() which:

1. Validates the opcode and flags.
2. Captures credentials (saves current cred for async use).
3. Calls the operation-specific preparation function (e.g., io_read_prep for IORING_OP_READ).
4. Attempts an inline (synchronous) completion: if the operation completes immediately (e.g., the data is already in the page cache), it posts the CQE right there without going async.
5. If the operation cannot complete immediately, it queues the request to a worker thread pool or the block layer's async infrastructure.

Completion posting: io_req_complete When an async operation finishes (in interrupt context, a workqueue, or inline), io_req_complete() is called. It writes the CQE to the CQ ring (using io_cqring_add_event), then calls io_cqring_ev_posted() which wakes up any threads blocked in io_uring_enter waiting for completions.

The SQPOLL kernel thread With IORING_SETUP_SQPOLL, io_sq_thread() runs as a kernel thread. It spins calling io_submit_sqes in a tight loop, checking the SQ tail for new entries. It uses io_ring_ctx->sq_wait to sleep when idle and io_ring_sq_wake() (called from io_uring_enter) to wake it.

io_ring_ctx — the central structure This is worth looking up in the source (include/linux/io_uring_types.h or io_uring/io_uring.c). It contains: the SQ/CQ rings, the registered buffers/files arrays, cancellation hashes, the task_work list for completion in task context, NUMA topology info, and statistics counters. Understanding this structure is the first step to writing a new io_uring operation.

/* Simplified kernel flow (pseudocode matching real structure names) */

/* sys_io_uring_enter -> */
long io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,
                    unsigned min_complete, unsigned flags) {
    if (to_submit)
        io_submit_sqes(ctx, to_submit);

    if (flags & IORING_ENTER_GETEVENTS)
        io_cqring_wait(ctx, min_complete); /* blocks if needed */
    return 0;
}

void io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) {
    for (unsigned i = 0; i < nr; i++) {
        struct io_uring_sqe *sqe = get_next_sqe(ctx);
        struct io_kiocb *req = io_alloc_req(ctx);
        io_init_req(ctx, req, sqe);  /* capture creds, opcode, etc */
        io_submit_sqe(ctx, req);     /* attempt inline or queue async */
    }
}

/* When async op finishes (e.g. in block layer callback): */
void io_req_complete(struct io_kiocb *req, s32 res) {
    struct io_ring_ctx *ctx = req->ctx;
    struct io_uring_cqe *cqe = io_get_cqe(ctx); /* next CQE slot */
    cqe->user_data = req->user_data;
    cqe->res = res;
    cqe->flags = 0;
    /* Advance CQ tail (with memory barrier) */
    smp_store_release(&ctx->rings->cq.tail,
                      ctx->rings->cq.tail + 1);
    io_cqring_ev_posted(ctx); /* wake waiters */
}

io_uring is not just a faster way to do read/write. It is a unified async interface for nearly everything the kernel can do. The current operation set (as of Linux 6.x) includes over 60 opcodes, and the list grows with each kernel release.

File I/O: IORING_OP_READ, IORING_OP_WRITE, IORING_OP_READ_FIXED, IORING_OP_WRITE_FIXED, IORING_OP_READV, IORING_OP_WRITEV, IORING_OP_FSYNC, IORING_OP_FALLOCATE, IORING_OP_FADVISE, IORING_OP_MADVISE, IORING_OP_OPENAT, IORING_OP_OPENAT2, IORING_OP_CLOSE, IORING_OP_STATX, IORING_OP_UNLINKAT, IORING_OP_RENAMEAT, IORING_OP_MKDIRAT.

Networking: IORING_OP_ACCEPT, IORING_OP_CONNECT, IORING_OP_RECV, IORING_OP_SEND, IORING_OP_RECVMSG, IORING_OP_SENDMSG, IORING_OP_SENDTO, IORING_OP_SOCKET, IORING_OP_BIND, IORING_OP_LISTEN, IORING_OP_SETSOCKOPT, IORING_OP_GETSOCKOPT.

Control: IORING_OP_POLL_ADD, IORING_OP_POLL_REMOVE, IORING_OP_TIMEOUT, IORING_OP_TIMEOUT_REMOVE, IORING_OP_LINK_TIMEOUT, IORING_OP_CANCEL, IORING_OP_ASYNC_CANCEL.

Advanced: IORING_OP_SPLICE, IORING_OP_TEE (zero-copy pipe operations), IORING_OP_PROVIDE_BUFFERS, IORING_OP_REMOVE_BUFFERS, IORING_OP_EPOLL_CTL (manage epoll from io_uring), IORING_OP_WAITID, IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAKE.

The new paradigm The existence of IORING_OP_FUTEX_WAIT and IORING_OP_WAITID is particularly interesting: it means you can integrate process synchronization and child process waiting into your io_uring event loop. Your entire application can be driven from a single io_uring loop — no separate signal handling, no separate epoll fd, no separate timeout thread. This is the vision: one ring to rule all asynchronous operations.

Language runtimes are starting to adopt this. Tokio (Rust's async runtime) has io_uring support via the io-uring crate. Node.js has experimental io_uring support for file I/O. Java's Project Loom and newer JDKs use io_uring on Linux. Understanding io_uring puts you at the foundation of how modern async programming actually works on Linux.

/* A taste of the full operation diversity */

/* Zero-copy splice between two fds (no userspace buffer needed) */
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_splice(sqe,
    pipe_in_fd,  -1,  /* read from pipe */
    socket_fd,   -1,  /* write to socket */
    4096,             /* bytes */
    SPLICE_F_MOVE);

/* Async futex wait: block on a userspace futex in the ring loop */
uint32_t futex_val = 0;
sqe = io_uring_get_sqe(&ring);
io_uring_prep_futex_wait(sqe,
    &futex_val,       /* futex address */
    0,                /* wait while value == 0 */
    FUTEX_BITSET_MATCH_ANY,
    FUTEX2_SIZE_U32,  /* flags */
    0);
sqe->user_data = FUTEX_WAIT_TAG;

/* Async openat (non-blocking open!) */
sqe = io_uring_get_sqe(&ring);
io_uring_prep_openat(sqe, AT_FDCWD,
    "/var/log/app.log",
    O_RDWR | O_CREAT | O_APPEND,
    0644);
sqe->user_data = OPEN_TAG;
/* No blocking open() call needed — result comes as CQE */