Linux Block Layer & IO Scheduling Deep Dive: From blk-mq Multi-Queue to BFQ/Kyber Schedulers and NVMe Optimization

Tuning via sysfs:

/sys/block/sda/queue/iosched/read_expire    (default 500ms)
/sys/block/sda/queue/iosched/write_expire   (default 5000ms)
/sys/block/sda/queue/iosched/writes_starved  (default 2)
/sys/block/sda/queue/iosched/fifo_batch      (default 16)
/sys/block/sda/queue/iosched/front_merges    (default 1)

Kyber: Target-Latency Feedback Control

Kyber (introduced in Linux 5.0) takes a fundamentally different approach: it uses a feedback controller that tracks average request latency per scheduling domain and throttles dispatches to meet target latencies.

// Kyber domains
enum kyber_domain {
    KYBER_READ,       // Read latency target (default: 2ms)
    KYBER_WRITE,      // Write latency target (default: 10ms)
    KYBER_DISCARD,    // Discard target (default: 5ms)
    KYBER_OTHER,      // FLUSH/SECURE_ERASE (default: 25ms)
};

struct kyber_queue_data {
    struct kyper_latency_depth kqd[KYBER_DOMAINS];
    spinlock_t lock;
};

Key concepts:

• Token bucket throttle: Each domain has a dispatch depth limit (tokens). Requests dispatch reduces tokens; completions restore tokens based on target latency divided by measured latency.
• Self-tuning: If average latency exceeds the target, the throttle reduces (fewer concurrent requests allowed). If latency is below target, throttle loosens to allow more parallelism.
• Request batch fairness: Within the same scheduling domain, requests are FIFO (no reordering). Work-conserving — never leaves HW idle when requests are pending.

This makes Kyber particularly effective for NVMe SSDs:

• No rotational delay considerations (only NAND flash read/write latency)
• Fast feedback loop — NVMe completes in microseconds, not milliseconds
• Adaptive concurrency — automatically scales depth with workload intensity

BFQ (Budget Fair Queueing): Proportional Share

BFQ (also backported from CFQ, introduced in 4.12) is the most sophisticated scheduler. It implements fair-bandwidth distribution across processes/cgroups while maintaining low latency:

struct bfq_data {
    struct rb_root service_tree;      // Sorted by virtual time
    struct bfq_entity *active_entity; // Currently dispatching entity
    u64 budget;                        // Current entity's budget
    unsigned long wr_coeff;             // Write compensation factor
    struct bfq_group *active_group;      // For group scheduling
    u64 last_finish_sects;              // For next-budget heuristic
    u64 min_budget;                     // Minimum budget per dispatch
}

BFQ's scheduling model uses budgets + virtual time:

1. Budget per dispatch: Each process is allocated a budget (in sectors or I/O operations) for each dispatch. Larger budgets mean more I/O per dispatch, but potentially higher latency for others.
2. Budget adaptation — low-latency mode: For interactive workloads (low-latency switches), BFQ reduces budgets aggressively so each process gets only a small I/O slice — keeping response times bounded at the cost of throughput.
3. Idle injection: If a process needs sequential throughput but waiting for its next budget grant would stall, BFQ injects idle time (~1ms) to allow the next process to complete its budget, then returns to the first. This "early completion heuristic" keeps both processes happy.
4. Write compensation: Writes are considered 10x more expensive than reads by default (configurable via bfq_wr_coeff) because writes have higher impact on dirty page cache pressure.

For cgroup-based proportional share, BFQ operates on bfq_group entities instead of processes, distributing budgets according to io.bfq.weight cgroup values.

None (Noop): Direct Dispatch

The none scheduler (also called noop for non-mq) performs minimal processing:

• Only performs front/back merging — no sorting
• FIFO dispatch to hardware
• Appropriate for fast storage where hardware handles queuing (NVMe SR-IOV, hardware RAID, cloud hypervisor paravirtual)
• Often used with io_uring fixed files where user-space controls submission directly

Scheduler Selection Decision Matrix

4. NVMe Passthrough: Bypassing the Block Layer

For extreme performance applications, the Linux block layer and even blk-mq may introduce unacceptable overhead. NVMe Passthrough allows user-space applications to submit NVMe commands directly to the device, bypassing the kernel block layer entirely.

NVMe ioctl Passthrough

The NVME_IOCTL_SUBMIT_IO_CMD ioctl (defined in linux/nvme_ioctl.h) provides direct NVMe command submission from user-space:

struct nvme_user_io {
    __u8  opcode;
    __u8  flags;
    __u16 control;
    __u16 nblocks;
    __u16 rsvd;
    __u64 metadata;
    __u64 addr;
    __u64 slba;
    __u32 dsmgmt;
    __u32 reftag;
    __u16 apptag;
    __u16 appmask;
};

ioctl(fd, NVME_IOCTL_SUBMIT_IO_CMD, &nvme_user_io);

Limitations of ioctl passthrough:

• One command per ioctl call — high syscall overhead (~1-2us per op)
• Uses O_DIRECT alignment (logical block size alignment required)
• No queuing — one command at a time

io_uring NVMe Passthrough (uring_cmd)

Linux 5.19+ introduced io_uring uring_cmd — a framework for drivers to register uring command operations, enabling batched, queued passthrough submissions via the high-performance io_uring interface:

// Register NVMe uring_cmd (kernel-side)
static const struct file_operations nvme_ctrl_fops = {
    .owner = THIS_MODULE,
    .uring_cmd = nvme_ctrl_ioctl_async,  // uring_cmd handler
    .uring_cmd_ops = {
        .flags =  URING_CMD_FIXED_BUFFERS,
        .iov_fixed_array = ...,
    },
};

// User-space submission (one SQE, multiple commands in batch)
struct nvme_uring_cmd new_cmd = {
    .opcode = nvme_cmd_write,
    .nsid = 1,
    .cdw10 = lba & 0xffffffff,
    .cdw11 = lba >> 32,
    .cdw12 = length - 1,
    .addr = (__u64)buf,
    .data_len = length * 512,
    .cdw13 = 0,
    .timeout_ms = 30000,
};
sqe->cmd_op = nvme_uring_cmd;
memcpy(sqe->cmd, &new_cmd, sizeof(new_cmd));
io_uring_submit(ring);  // Batch submit all SQEs at once

Key advantages of uring_cmd NVMe passthrough vs. regular block layer:

• Zero syscall cost: SQPOLL mode eliminates io_uring_enter entirely
• Batched submission: Multiple commands submitted in a single batch
• Dual queues architecture: Block layer and uring_cmd queues allow concurrent use
• Fixed buffers, fixed files: Optional persistent resource binding eliminates per-op overhead
• Use cases: SPDK, NVMe-oF initiators, RAID controllers, custom filesystem engines

5. io_uring and Block Layer Integration

io_uring's interaction with the block layer is one of its most powerful features for storage workloads. Three key integration points deserve special attention:

Fixed Buffers for Block I/O

When using IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED, the user pre-registers buffers with io_uring (IORING_REGISTER_BUFFERS) and the block layer can map these buffers directly — eliminating per-operation pin/unpin overhead:

// - Without fixed buffers: each op = get_user_pages + map + read/write + unmap + put_user_pages
// - With fixed buffers:     each op = read/write (buffer already pinned and mapped)

For database workloads (PostgreSQL, MySQL with O_DIRECT), fixed buffers reduce per-operation overhead by 15-20% at high IOPS rates.

Polling Mode (IORING_SETUP_IOPOLL)

For io_uring block operations, enabling IORING_SETUP_IOPOLL causes completions to be polled rather than interrupt-driven:

• Spin in kernel on CQ head pointer update
• No IRQ context, no softirq, no scheduling delay
• Completes in <1us vs ~10-50us interrupt-based
• Trade-off: CPU spinning vs latency savings
• Use case: Ultra-low-latency NVMe applications (high-frequency trading, real-time databases)

Application-Layer I/O Priorities

io_uring SQEs support per-operation I/O priority via IOSQE_BUFFERED flags and IOSQE_FIXED_FILE priority context. Combined with blk-mq's priority mapping:

enum ioprio_class {
    IOPRIO_CLASS_NONE = 0,
    IOPRIO_CLASS_RT = 1,    // Real-time (levels 0-7)
    IOPRIO_CLASS_BE = 2,    // Best-effort
    IOPRIO_CLASS_IDLE = 3,  // Only when no other I/O
};

io_uring applications can set per-SQE I/O priority via sqe->ioprio before submission — enabling fine-grained priority control without process-wide ioprio_set() system call changes.

6. Production Performance Tuning Guide

Hardware & Queue Depth

# NVMe SSD: max queue depth and poll queues
echo 256 > /sys/block/nvme0n1/queue/nr_requests

# Enable polling for NVMe (reduces IRQ overhead for high-IOPS)
echo 1 > /sys/block/nvme0n1/queue/io_poll

# Set poll queue count (per-NUMA-node polling contexts)
echo 4 > /sys/block/nvme0n1/queue/io_poll_delay

# Enable write-back cache flush
echo 1 > /sys/block/nvme0n1/queue/write_cache

I/O Scheduler Selection

# For NVMe database workloads (minimal overhead):
echo none > /sys/block/nvme0n1/queue/scheduler

# For mixed-read-write SSD workloads (bounded tail latency):
echo mq-deadline > /sys/block/nvme0n1/queue/scheduler

# For fairness across containers/tenants on fast storage:
echo bfq > /sys/block/nvme0n1/queue/scheduler

Read-Ahead and Merging

# Read-ahead: set to 256-512 sectors for sequential workloads (128-256KB)
echo 256 > /sys/block/nvme0n1/queue/read_ahead_kb

# Max sectors per request: increase for sequential workloads
echo 1024 > /sys/block/nvme0n1/queue/max_sectors_kb

# Enable merging in scheduler
echo 1 > /sys/block/nvme0n1/queue/nomerges  (0=normal, 1=no_merges, 2=no_merges_or_sort)

NUMA-Aware Tuning

// When using multiple NVMe devices across NUMA nodes
// Set hw_queue affinity to local NUMA node
echo 0 > /sys/block/nvme0n1/queue/rq_affinity  // 0=none, 1=local, 2=prefer_local

// For io_uring: register fixed buffers on the same NUMA node as device
// Create per-NUMA-node io_uring instances, bind vCPUs to local node

7. Real-World Performance Benchmarks

Benchmark results from a 4-core VM on an NVMe SSD (Samsung PM9A3, 7000MB/s read, 600MB/s write, 1M IOPS max):

Observations:

• Scheduler selection matters 5-15% at high IOPS but 2-5x difference in tail latency
• io_uring + IOPOLL provides 40% better IOPS with 80% lower tail latency vs sync block I/O
• NVMe passthrough via uring_cmd achieves near-line-rate IOPS at 3.5x lower latency
• All configurations CPU-bound before IOPS-bound — 2 cores saturates PM9A3's physical limits

8. Modern Developments and Future Directions

Block Layer Writeback Throttling (wbt)

Writeback throttling (wbt) controls the number of dirty pages the block layer will queue for writeback before throttling submitting processes. This prevents OOM conditions and excessive disk utilization:

/sys/block/sda/queue/wbt_lat_us  (default: 2000µs)
echo 1000 > /sys/block/sda/queue/wbt_lat_us  // Aggressive throttle
echo 5000 > /sys/block/sda/queue/wbt_lat_us  // Relaxed

blk-cgroup I/O Throttling

The block layer integrates directly with cgroups v2 for I/O resource control:

// io.max limits
echo "nvme0n1 rbps=1073741824 wbps=536870912 riops=100000 wiops=50000" > /sys/fs/cgroup/mycgroup/io.max

// io.weight proportional share
echo 100 > /sys/fs/cgroup/mycgroup/io.weight  // Default 100, relative to siblings

// io.latency latency target enforcement
echo "nvme0n1 target=100000" > /sys/fs/cgroup/mycgroup/io.latency  // 100ms target

Recent Kernel Developments (6.x)

• blk-mq support for zone block devices (ZNS, SMR): Adds blk_queue_max_active_zones() and blk_queue_max_open_zones() — critical for Zoned Namespaces SSDs used in Ceph and distributed object stores
• io_uring block layer inline completion (6.5+): Small reads (<4KB) can complete synchronously within the SQE submission path, bypassing CQ polling entirely
• blk-mq rq_attrs_size: Driver-extended per-request attributes enable richer hardware descriptors without struct bloat
• blk-cgroup v2 cgroup-bpf: BPF programs can now inspect and modify block I/O at the cgroup level for custom fairness schemes

Conclusion

The Linux block layer is a masterpiece of kernel engineering — balancing performance, robustness, and flexibility across a spectrum from spinning HDDs to ultra-low-latency NVMe. The blk-mq multi-queue architecture solved the scaling crisis of the 2010s, while io_uring NVMe passthrough continues to push the boundaries of what is achievable without custom hardware.

For system engineers, understanding these layers is essential: the I/O scheduler you choose can be the difference between 50µs and 5ms tail latency; enabling io_uring with IOPOLL can double your IOPS at half the CPU; and proper NUMA-affine io_uring configuration can extract the last 10-15% of performance from your storage infrastructure. The block layer remains one of the most active areas of kernel development, with NVMe, ZNS, io_uring, and cgroup-bpf integrations continuing to reshape how applications interact with persistent storage.