Linux块设备层与I/O调度器深度实战：从blk-mq到Kyber/BFQ选型指南

The Linux Block Layer and I/O Scheduler Architecture

The Linux block layer is the fundamental infrastructure that mediates between user-space applications and storage devices. Understanding its internals—particularly the I/O scheduler subsystem—is essential for anyone working on storage performance optimization, database engineering, or low-latency application design.

Block Layer Fundamentals

The bio and request Structure

At the heart of the block layer lies the struct bio, which represents an ongoing I/O operation. Each bio contains a list of segments (collections of pages), a device reference, and operation flags (READ, WRITE, FLUSH, DISCARD). The bio is the universal currency of I/O in the kernel—every filesystem request eventually decomposes into one or more bios before reaching the driver.

The request structure groups multiple bios into a single hardware submission unit, tagged with a request queue. Modern NVMe drives support deep queues (up to 64K entries per namespace), and efficient request merging is critical for throughput.

The Multi-Queue Block Layer (blk-mq)

Introduced in Linux 3.13 and now mandatory for all modern storage devices, the blk-mq framework eliminates the single global request queue bottleneck. It introduces a two-tier queue architecture:

• Hardware Dispatch Queue (hctx): One per CPU or NUMA node, mapped directly to hardware submission queues. NVMe devices with multiple queues can achieve near-linear scaling with core count.
• Software Queue (ctx): Per-CPU staging area where the I/O scheduler operates before handing requests off to hardware queues.

This design eliminates lock contention and allows the storage stack to sustain millions of IOPS on modern NVMe hardware.

I/O Schedulers: Algorithms and Use Cases

mq-deadline

mq-deadline is the default scheduler for most SCSI/SATA devices and many NVMe drives. It implements a straightforward algorithm: read requests are prioritized over writes (since reads block the caller), and operations have expiration deadlines to prevent starvation.

Key tunables: read_expire (500ms default), write_expire (5000ms default), writes_starved (2 by default), fifo_batch (16 by default). Latency-sensitive workloads (OLTP databases) often tune these—reducing write_expire to force faster write flushing without pacing the device during bursty sequential writes.

Kyber

Karizalios, McKenney, and Bergman designed Kyber as a latency-target scheduler rather than a throughput-maximizer. It tracks request latency and dynamically adjusts dispatch rates to meet configurable targets (read_lat_nsec, write_lat_nsec). This is ideal for mixed workloads where P99 latency matters more than average throughput.

Kyber's algorithm works by self-tuning: if observed latency exceeds the target, it throttles new dispatches, waiting for in-flight requests to complete. If latency is below target, it dispatches more aggressively.

BFQ (Budget Fair Queueing)

BFQ is designed for interactive systems and latency-sensitive applications. It guarantees I/O bandwidth to each process proportional to its "budget," and uses a sophisticated prediction model to anticipate process I/O patterns. BFQ excels for desktops and mixed-use servers but can suffer from high CPU overhead under extreme IOPS loads.

Tunable parameters include low_latency (on by default), which trades throughput for predictability, and per-service weights tunable via ioprio classes.

none (No-op / Passthrough)

For modern NVMe drives with minimal seek latency, many workloads benefit from simply bypassing software scheduling entirely. The none scheduler dispatches bios directly from per-CPU software queues into hardware submission queues, eliminating scheduling overhead entirely. This is often the default recommendation for in-memory databases (Redis), NVMe-backed Kubernetes storage, and applications that implement their own I/O scheduling.

Advanced Techniques and Tuning

I/O Merging and QoS

Adjacent sector bios are merged in the scheduler layer before dispatch, reducing the number of device commands. For QoS-sensitive deployments, the blk-mq framework supports blk-wbt (write throttling) and ionice-based priority classes.