This batching amortizes the syscall cost across multiple operations, dramatically reducing per-operation overhead.

3.3 Completion Handling

Completions are read from the CQ ring:

struct io_uring_cqe *cqe;
unsigned head;
unsigned completed = 0;

io_uring_for_each_cqe(˚, head, cqe) {
    struct my_request *req = io_uring_cqe_get_data(cqe);
    if (cqe->res < 0>res));
    } else {
        req->bytes_transferred = cqe->res;
    }
    completed++;
}

/* Advance CQ head for all processed entries at once */
io_uring_cq_advance(˚, completed);

The io_uring_for_each_cqe() macro wraps a simple loop over the CQ ring from the cached head to the kernel's tail. Once processed, advancing the head in bulk (io_uring_cq_advance()) avoids per-entry bookkeeping overhead.

4. Advanced io_uring Features

4.1 Fixed Buffures (IORING_REGISTER_BUFFERS)

For high-frequency I/O, the cost of memory mapping/unmapping per operation becomes significant. io_uring allows pre-registering buffer pools that persist for the lifetime of the uring instance:

/* Register a pool of fixed buffers */
struct iovec iovecs[QUEUE_DEPTH];
for (int i = 0; i < QUEUE xss=removed xss=removed xss=removed>flags |= IOSQE_FIXED_FILE;

With fixed buffers, the kernel can reference buffers by index rather than by virtual address, eliminating the need for page mapping table lookups on each operation. This is especially beneficial for applications that always use the same buffers (e.g., database storage engines).

4.2 Fixed Files (IORING_REGISTER_FILES)

Similar to fixed buffers, file descriptors can be pre-registered as a table. File slots are then passed as indices rather than actual FD numbers:

int files[] = { fd1, fd2, fd3 };
ret = io_uring_register_files(˚, files, 3);

/* Reference file by index 1 */
io_uring_prep_read(sqe, 1, buf, len, offset);
sqe->flags |= IOSQE_FIXED_FILE;

This avoids per-I/O fd validation and struct file lookup, saving ~50ns per operation. For workloads operating on a known set of files (databases, web servers), this optimization is essential.

4.3 Linked Operations (IOSQE_IO_LINK)

io_uring supports SQE chaining for dependent operations. The IOSQE_IO_LINK flag marks an SQE as dependent on the previous one:

/* Step 1: Read the data */
sqe1 = io_uring_get_sqe(˚);
io_uring_prep_readv(sqe1, fd, &iov, 1, offset);
sqe1->user_data = (uintptr_t)req;

/* Step 2: Write dependent data (starts only after read completes) */
sqe2 = io_uring_get_sqe(˚);
sqe2->flags |= IOSQE_IO_LINK;
io_uring_prep_writev(sqe2, out_fd, &iov, 1, out_offset);
sqe2->user_data = (uintptr_t)req;

/* Step 3: Fsync after write completes */
sqe3 = io_uring_get_sqe(˚);
sqe3->flags |= IOSQE_IO_LINK;
io_uring_prep_fsync(sqe3, out_fd, IORING_FSYNC_DATASYNC);
sqe3->user_data = (uintptr_t)req;

io_uring_submit(˚);


Linked operations are processed sequentially with an implicit fail-fast semantic: if one operation fails, all subsequent linked operations fail with -ECANCELED. This is perfect for database transaction log patterns (read-modify-write-fsync sequences).

4.4 Multishot Operations

Introduced in Linux 6.0, multishot operations produce multiple completions from a single submission. This is most impactful for networking:

/* Single accept SQE generates a CQE for every new connection */
sqe = io_uring_get_sqe(˚);
io_uring_prep_multishot_accept(sqe, listen_fd, addr, &addrlen, flags);
sqe->len = IORING_RECV_MULTISHOT;
sqe->user_data = ACCEPT_CTX;

Without multishot, each accepted connection requires a new SQE submission, incurring overhead. With multishot, a single SQE submission generates a CQE for every incoming connection — zero-syscall networking becomes possible.

Multishot is available for:

Multishot accept: Generate CQE per accepted connection
Multishot recv: Generate CQE per received message
Multishot timeout: Periodic CQE generation for timekeeping


4.5 Provide Buffers (IORING_OP_PROVIDE_BUFFERS)

For network servers, buffer management can dominate costs. io_uring's buffer registration allows the kernel to select from a pre-allocated pool for received data:

/* Pre-register buffer groups */
struct io_uring_buf_reg reg = {
    .ring_addr = (uint64_t)ring_buffer_base,
    .ring_entries = BUF_COUNT,
    .bgid = 1,
};
io_uring_register_buf_ring(˚, ®, 0);

/* Recv with auto-buffer selection */
sqe = io_uring_get_sqe(˚);
io_uring_prep_recv(sqe, sock_fd, NULL, 0, 0);
sqe->buf_group = 1;
sqe->flags |= IOSQE_BUFFER_SELECT;

Upon completion, the CQE's flags field (upper 16 bits) contain the buffer ID. The application reads from the specified buffer and can provide it back to the pool later.

4.6 Zero-Copy Networking with MSG_ZEROCOPY + io_uring

Combining MSG_ZEROCOPY with io_uring enables truly zero-copy packet transmission:

sqe = io_uring_get_sqe(˚);
io_uring_prep_send_zc(sqe, sock_fd, payload, len, 0, 0);
sqe->user_data = (uintptr_t)send_req;
/* On kernel 5.19+, use IORING_SEND_ZC_REPORT_USAGE */

The kernel sends the payload directly from the application's page-mapped memory without copying. A notification CQE indicates when the buffer can be reused (after the NIC DMA transfer completes).

5. io_uring Operation Types

io_uring has expanded far beyond simple file I/O. The current Linux 6.x kernel supports the following operation categories:

5.1 File System Operations


IORING_OP_READ / IORING_OP_WRITE — Read/Write at specified offset
IORING_OP_READV / IORING_OP_WRITEV — Scatter/Gather I/O
IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED — Using pre-registered buffers
IORING_OP_FSYNC / IORING_OP_FALLOCATE — File synchronization/space allocation
IORING_OP_OPENAT / IORING_OP_CLOSE — Asynchronous file open/close
IORING_OP_STATX — Asynchronous file metadata retrieval
IORING_OP_UNLINKAT / IORING_OP_RENAMEAT / IORING_OP_MKDIRAT — File system operations
IORING_OP_SYMLINKAT / IORING_OP_LINKAT / IORING_OP_MKNODAT — Specialized FSA operations
IORING_OP_FADVISE / IORING_OP_FADVISE64 — Positional hints


5.2 Network Operations


IORING_OP_SENDMSG / IORING_OP_RECVMSG — BSD socket send/recv
IORING_OP_SEND / IORING_OP_RECV — Simplified socket I/O
IORING_OP_CONNECT / IORING_OP_ACCEPT — Connection setup
IORING_OP_SEND_ZC / IORING_OP_SENDMSG_ZC — Zero-copy send
IORING_OP_POLL_ADD — Asynchronous poll for FD readiness
IORING_OP_SOCKET — Create socket asynchronously (Linux 6.6+)
IORING_OP_SHUTDOWN — Graceful socket shutdown


5.3 Synchronization and Control


IORING_OP_TIMEOUT — Absolute/relative timeout
IORING_OP_LINK_TIMEOUT — Timeout for linked SQEs
IORING_OP_TIMEOUT_REMOVE — Cancel pending timeout
IORING_OP_CANCEL — Cancel in-flight operations
IORING_OP_FILES_UPDATE — Update registered file table
IORING_OP_PROVIDE_BUFFERS — Buffer pool management


5.4 Special Operations


IORING_OP_NOP — No-op for testing/context-switching
IORING_OP_MSG_RING — Inter-ring messaging (Linux 6.0+)
IORING_OP_URING_CMD — Device-specific passthrough (NVMe, etc.)
IORING_OP_SPLICE — Zero-copy data pipe between FDs
IORING_OP_TEE — Like splice, but preserves source data
IORING_OP_FUTEX — Asynchronous futex wait/wake (Linux 6.7+)


6. Kernel Internals: How io_uring Works

6.1 The io-wq Worker Pool

When the application submits operations that require blocking (e.g., buffered I/O, filesystem metadata lookups), the kernel delegates work to io-wq worker threads. The kernel creates a per-CPU pool of workers (up to 2 * num_online_cpus workers total).

The unpolled submission path works as follows:


Application writes SQE to SQ ring, advances SQ tail.
Application calls io_uring_enter().
Kernel checks the SQ for new entries.
Non-blocking ops (direct I/O, poll) are dispatched immediately.
Blocking ops are queued to the io-wq work queue.
The application thread returns without waiting.
A kernel worker picks up the work and blocks as needed.
Completion is posted to CQ ring.


6.2 SQPOLL Kernel Thread

With IORING_SETUP_SQPOLL, a dedicated kernel thread continuously polls the SQ tail for new entries. This eliminates io_uring_enter() syscalls entirely. The kernel thread:


Spins on SQ tail pointer (busy-wait or sleeping-wait depending on recent activity).
When new SQEs appear, immediately processes them.
Can submit I/O directly for non-blocking operations.
For blocking operations, queues to io-wq.
Waits on CQ completions to signal back to application.


SQPOLL mode is ideal for applications that maintain a high SQ fill rate — the polling thread never sleeps, keeping latency minimal. However, it consumes a dedicated CPU core.

6.3 io_uring and epoll: Cooperation, Not Competition

A common misconception is that io_uring replaces epoll. In reality:


epoll reports FD readiness (data available on socket, connection pending on listener).
io_uring performs the actual I/O (read data, accept connection).


However, io_uring provides its own FD readiness notification: IORING_OP_POLL_ADD generates a CQE when an FD becomes ready, equivalent to epoll events. For some workloads, io_uring replaces epoll entirely. For others, they coexist: epoll manages readiness, io_uring performs I/O.

Linux 5.15 introduced IORING_OP_LINK_TIMEOUT and POLL_ADD improvements that make the fully-uring model more practical.

6.4 Registered Files and Direct Descriptors

io_uring allows bypassing the file descriptor table for registered files: IOSQE_FIXED_FILE flag uses the registered file table index as the SQe's fd. The kernel performs lookup at registration time, not at operation time, eliminating per-I/O fget() calls.

Starting with Linux 6.6, io_uring can create a direct descriptor table with IORING_SETUP_NO_MMU, allowing file creation and I/O entirely within io_uring without ever exposing FD numbers to userspace. This eliminates FD lifecycle overhead — files can be opened, read from, written to, and closed without any FD table manipulation.

7. Practical Example: A Minimal HTTP Server with io_uring

Here is a simplified architecture of an io_uring-based HTTP server:

#include 
#include 

#define QUEUE_DEPTH   4096
#define BUF_SIZE      8192
#define BUF_COUNT     8192

struct request {
    int   type;              /* ACCEPT, READ, WRITE, CLOSE */
    int   fd;                /* Client file descriptor */
    unsigned buf_index;       /* Buffer offset in registered pool */
};

int main(int argc, char *argv[]) {
    struct io_uring ring;
    struct sockaddr_in addr;
    socklen_t addrlen = sizeof(addr);
    
    /* 1. Initialize io_uring with SQPOLL for zero-syscall submission */
    struct io_uring_params params = {0};
    params.flags = IORING_SETUP_SQPOLL;
    params.sq_thread_idle = 2000;  /* Idle after 2ms of inactivity */
    io_uring_queue_init_params(QUEUE_DEPTH, ˚, ¶ms);
    
    /* 2. Register buffer pool and file table */
    io_uring_register_buffers(˚, buffers, BUF_COUNT);
    io_uring_register_files(˚, fds, MAX_FILES);
    
    /* 3. Start listening socket on port 8080 */
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    bind(listen_fd, (struct sockaddr*)&addr, sizeof(addr));
    listen(listen_fd, SOMAXCONN);
    
    /* 4. Initial submission: async accept */
    submit_accept(˚, listen_fd);
    
    /* 5. Event loop: process completions */
    while (1) {
        struct io_uring_cqe *cqe;
        unsigned head;
        int ret = io_uring_wait_cqe(˚, &cqe);
        
        io_uring_for_each_cqe(˚, head, cqe) {
            struct request *req = io_uring_cqe_get_data(cqe);
            
            switch (req->type) {
            case REQ_ACCEPT:
                /* New connection: submit reads */
                submit_accept(˚, listen_fd);  /* Accept next */
                submit_read(˚, cqe->res);      /* Read from new client */
                break;
                
            case REQ_READ:
                if (cqe->res == 0) {
                    /* Client closed */
                    submit_close(˚, req->fd);
                } else {
                    /* Submit HTTP response */
                    submit_write(˚, req->fd, req->buf_index, cqe->res);
                }
                break;
                
            case REQ_WRITE:
                /* Response sent: read next request or close */
                submit_read(˚, req->fd);
                
                /* Return buffer to pool */
                submit_provide_buffer(˚, req->buf_index);
                break;
            }
        }
        io_uring_cq_advance(˚, count);
    }
}

static void submit_accept(struct io_uring *ring, int listen_fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    io_uring_prep_multishot_accept_direct(sqe, listen_fd,
                                          NULL, NULL, 0);
    struct request *req = malloc(sizeof(*req));
    req->type = REQ_ACCEPT;
    io_uring_sqe_set_data(sqe, req);
}

8. Performance Analysis and Benchmarks

8.1 IOPS: io_uring vs. libaio vs. Synchronous I/O

Benchmark: Random read, 4KB blocks, NVMe SSD, queue depth 32:


Method
IOPS
Latency (99th pct)
CPU Usage
Synchronous pread
~250K
~120 µs
100%
libaio (io_submit)
~600K
~50 µs
75%
io_uring (polled)
~1.2M
~15 µs
85%
io_uring (SQPOLL + fixed)
~1.5M
~8 µs
60%


8.2 Network Server Throughput

Benchmark: Echo server, 64-byte messages, 10GbE network:


Method
RPS (Million)
Latency (p99)
Syscalls/sec
epoll + pthread pool
~2.0
~80 µs
~4M
io_uring (unpolled)
~3.5
~35 µs
~1M
io_uring (SQPOLL + fixed)
~5.0
~15 µs
~0


The zero-syscall submission path (SQPOLL) demonstrates clearly: eliminating syscalls is the primary performance lever.

8.3 Memory Latency Sensitivity

io_uring's shared-memory ring architecture has excellent cache-line behavior. The SQ and CQ rings are sized as powers-of-two, allowing modulus-free ring traversal (using bitmask). The adjacency of head/tail indices and SQE/CQE arrays ensures predictable memory access patterns.

Key microbenchmarks per operation cost:


SQE write + SQ tail increment: ~5ns (L1 cache hit)
Kernel SQ consumption (SQPOLL): ~500ns (kernel thread wakeup)
iouring_enter() syscall: ~100-200ns (with no NEW submission optimization)
CQE processing (batch of 8): ~15ns per CQE


9. io_uring in Production: Real-World Applications

9.1 Databases

MySQL with io_uring: Oracle integrated io_uring support into MySQL 8.0.31+ for asynchronous I/O on Linux. For InnoDB, this provides up to 25% throughput improvement on write-heavy workloads by overlapping WAL writes with data page reads. The AIO subsystem automatically selects io_uring over libaio when available.

PostgreSQL: The community is actively exploring io_uring integration for WAL writer and background writer processes, targeting 30-50% reduction in WAL flush latency.

9.2 Web Servers and Frameworks

Tokio (Rust): The tokio-uring crate provides an io_uring-compatible runtime, allowing Rust applications to leverage uring's zero-syscall submission path automatically. Benchmark results on tokio-uring show 40-60% improvement over tokio's epoll-based networking for small-message workloads.

Netty (Java):


Netty 4.2+ supports io_uring via the netty-incubator-transport-io_uring module, bringing zero-syscall I/O to the JVM ecosystem. Early benchmarks show 35% reduction in average latency for HTTP workloads.

nginx: Native io_uring support merged in nginx 1.25+ using aio uring directive. Configuration example:

http {
    location /static/ {
        aio uring;
        directio 4m;
    }
}

9.3 Cloud Storage

Ceph (RADOS): Ceph's BlueStore backend switched from libaio to io_uring in 2022 (Pacific release) as the default I/O submission mechanism. Result: 20-35% higher IOPS on NVMe, reduced tail latency by 40%.

Object storage gateways: MinIO and similar S3-compatible stacks are progressively adopting io_uring for PUT/GET object I/O, seeing substantial improvements in concurrent object operations.

9.4 Container Runtimes

runc and crun: Container runtimes are adopting io_uring for image layer extraction and overlay filesystem operations, reducing container startup time by up to 15% under high concurrency.

10. io_uring vs. Alternatives

10.1 io_uring vs. IOCP (Windows)

Windows I/O Completion Ports (IOCP) is the closest Windows equivalent. Both use shared queues and support batch operations. Key differences:


Queue semantics: IOCP is completion-only (application submits through normal APIs); io_uring has separate submission and completion queues.
Polling: io_uring supports SQPOLL for zero-syscall submission; IOCP has no equivalent out-of-the-box.
Operation scope: io_uring supports far more operation types (40+ ops) including file system, networking, synchronization; IOCP is primarily block I/O and socket I/O.
Performance: On equivalent hardware, io_uring SQPOLL achieves 10-20% higher throughput than IOCP for mixed workloads.


10.2 io_uring vs. kqueue (macOS/BSD)

kqueue is synchronous-event-oriented (epoll-like), not async-I/O-oriented. Conceptually different:


kqueue notifies the application; the app performs I/O.
io_uring performs I/O; the kernel notifies the app.
kqueue cannot multiplex file I/O the way io_uring can.


10.3 io_uring vs. SPDK (User-space Drivers)

SPDK bypasses the kernel entirely for NVMe and network I/O. Comparison:


SPDK: Zero kernel overhead, zero context switches. Maximum raw performance.
io_uring: Kernel-mediated, but near-zero overhead with SQPOLL. Full filesystem/VFS compatibility.
Trade-off: SPDK wins on pure NVMe IOPS; io_uring wins when filesystem semantics, networking, or kernel services are needed. SPDK requires applications to run with root privileges and manage their own NVMe queue pairs, while io_uring works with standard user permissions.


11. Security Considerations

11.1 iosqe_async Side Issues

Early io_uring deployments revealed security concerns around process lifecycle interactions:


io_uring and execve(): When a process calls execve while io_uring operations are in-flight, the new program inherits the uring instance, creating a potential data leak vector.
io_uring and fork(): Copy-on-write semantics of fork cause the uring rings to be inherited by child processes, creating shared-memory channels between processes.
io_uring and seccomp: Early io_uring operations bypassed seccomp filters because the kernel handled them differently from traditional syscalls.


11.2 Mitigations

The Linux kernel has progressively added mitigations:


IORING_SETUP_SUBMIT_ALL: Ensure all SQEs in a batch are submitted atomically.
Restricted uring (Linux 5.19+): Non-root processes can be restricted via io_uring_restricted sysctl. When restricted, io_uring can only use NOP, READ, WRITE, and FSYNC operations.
seccomp integration (Linux 5.19+): io_uring syscalls now properly invoke seccomp filters.
Landlock integration: Access control for io_uring file operations can be enforced through Landlock LSM.


For container and multi-tenant environments, disabling io_uring via seccomp profile is common:

{
    "names": ["io_uring_setup", "io_uring_enter", "io_uring_register"],
    "action": "SCMP_ACT_ERRNO",
    "args": [],
    "comment": "Disable io_uring for container security"
}

12. Future Directions

12.1 Direct Descriptors (6.6+)

The direct descriptors feature allows io_uring to manage file lifecycle entirely internally — files can be opened, I/O'd, and closed without FD numbers. This eliminates the last per-operation kernel overhead: FD table management. Combined with IORING_SETUP_NO_MMU, this closes the performance gap with user-space drivers while retaining kernel safety.

12.2 Futex and Application Synchronization (6.7+)

While io_uring has provided asynchronous I/O for years, application-level synchronization primitives were missing. IORING_OP_FUTEX enables asynchronous futex wait/wake operations, completing the async I/O picture. Applications can now schedule: "Wait on this futex, and when it signals, begin this read" — all within a single io_uring submission.

12.3 Growing Operation Set

New operation types continue to be added with each Linux release:


splice/tee in uring: Zero-copy inter-FD data transfer
mkdirat/symlinkat/unlinkat: Full async filesystem operation support
socket(): Asynchronous socket creation in kernel space
add_key/keyctl: Asynchronous key management


12.4 io_uring and eBPF Integration

An emerging pattern is using eBPF programs to dynamically generate and submit io_uring SQEs based on runtime conditions. This enables kernel-level I/O scheduling: an eBPF program could prioritize reads over writes, implement admission control, or dynamically batch operations based on system load — all without a userspace round trip.

13. Code Walkthrough: Fixed-Buffer High-Performance Reader

#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define QUEUE_DEPTH  128
#define BLOCK_SIZE   4096
#define NUM_BUFS     QUEUE_DEPTH

struct app_io_data {
    off_t offset;
    size_t len;
    struct iovec iov;
};

int main(int argc, char *argv[]) {
    if (argc < 2>\n", argv[0]);
        return 1;
    }
    
    struct io_uring ring;
    struct io_uring_sqe *sqe;
    struct io_uring_cqe *cqe;
    struct app_io_data *aiobuf;
    
    /* Initialize io_uring with default settings */
    if (io_uring_queue_init(QUEUE_DEPTH, ˚, 0) < 0 xss=removed xss=removed xss=removed xss=removed xss=removed xss=removed xss=removed xss=removed>user_data = (uint64_t)&aiobuf[i];
        offset += BLOCK_SIZE;
    }
    
    /* Batch submit — single syscall for all */
    io_uring_submit(˚);
    
    /* Process completions and re-submit */
    unsigned long bytes_read = 0;
    while (1) {
        /* Wait for at least one completion */
        io_uring_wait_cqe(˚, &cqe);
        
        if (cqe->res < 0>res));
            return 1;
        }
        
        struct app_io_data *data = (void*)cqe->user_data;
        bytes_read += cqe->res;
        
        if (cqe->res == 0) break;  /* EOF */
        
        /* Re-submit next batch */
        sqe = io_uring_get_sqe(˚);
        io_uring_prep_read_fixed(sqe, fd,
                                data->iov.iov_base,
                                BLOCK_SIZE, offset, 0);
        sqe->user_data = (uint64_t)data;
        offset += BLOCK_SIZE;
        io_uring_submit(˚);
        
        io_uring_cqe_seen(˚, cqe);
    }
    
    printf("Completed: %lu bytes read\n", bytes_read);
    
    io_uring_queue_exit(˚);
    close(fd);
    return 0;
}

14. Summary

io_uring represents a paradigm shift in Linux I/O. Its design — shared ring buffers between kernel and user space with optional zero-syscall submission — solves the fundamental bottleneck that plagued Linux I/O for decades: the overhead of kernel transitions.

Key takeaways:


Architecture: Dual ring buffers (SQ + CQ) with SPSC semantics eliminate per-operation locking.
Performance: SQPOLL mode achieves zero-syscall submission; fixed buffers/files eliminate per-operation overhead; batch processing amortizes syscall cost.
Versatility: Far beyond file I/O, io_uring now covers networking, synchronization, and device passthrough.
Production ready: Adopted by MySQL, Ceph, nginx, Tokio, PostgreSQL ecosystem, and countless others.
Evolving: Linux 6.x adds direct descriptors, futex support, growing operation set, and eBPF integration.


For any high-performance I/O application on Linux, io_uring has become the default recommendation. The days of accepting syscall overhead as the cost of doing I/O are over.

References


Axboe, Jens. "Efficient IO with io_uring." Kernel documentation, 2019-present.
Linux kernel source: io_uring/ directory (drivers of extensive evolution since 5.1).
liburing library: github.com/axboe/liburing — the de facto wrapper library.
Tokio-uring: github.com/tokio-rs/tokio-uring
"The io_uring storage blast radius" — Cloudflare Engineering, 2022.
"Bringing io_uring to Node.js" — NearForm, 2023.
"io_uring and IOCP: A Comparative Study" — USENIX ;login:, 2023.