Linux Kernel Interrupt Handling: From Hardware Exceptions to Softirq, a Deep Dive into the Complete Mechanism

Table of Contents

• Overview of the Interrupt Subsystem
• Hardware Foundation: Interrupt Controllers and IRQ Lines
• Interrupt Descriptor Table (IDT) and IDT Initialization
• Complete Path of Hardware Interrupt Handling
• Top-Half and Bottom-Half: The Origin of Interrupt Splitting
• Softirq: The Kernel's Low-Priority Interrupt Mechanism
• Tasklet: A Bottom-Half Mechanism Built on Softirq
• Workqueue: Bottom-Half Execution in Process Context
• Threaded IRQ: The Modern Approach to Interrupt Handling
• Timer Interrupts and Kernel Time Management
• Performance Monitoring and Tuning Tools
• Summary

1. Overview of the Interrupt Subsystem

Interrupts are the core mechanism by which the CPU responds to hardware events in modern operating systems. Whether it is a keyboard keystroke, a network packet arrival, or a disk I/O completion, all notifications to the CPU are delivered through interrupts. In the Linux kernel, the interrupt subsystem is a complex and critical component that spans hardware abstraction, context switching, scheduling, and concurrency control.

The entire interrupt handling flow can be divided into several key stages:

1. Hardware Interrupt Trigger: A peripheral device sends an interrupt signal to the interrupt controller via an IRQ line.
2. Interrupt Controller Routing: The interrupt controller (APIC/IO-APIC) routes the interrupt to a specific CPU core based on priority and affinity.
3. CPU Response: The CPU saves the current execution context and jumps to the corresponding interrupt handler.
4. Top-Half Handler Execution: Time-critical operations are executed in interrupt context, such as acknowledging the interrupt and reading hardware status.
5. Bottom-Half Handler Execution: Non-time-critical, time-consuming operations are deferred for later execution.
6. Context Restore: Restore the previously saved context and return to the pre-interrupt execution state.

2. Hardware Foundation: Interrupt Controllers and IRQ Lines

2.1 8259A PIC (Progrepesmable Interrupt Controller)

The early x86 architecture used the Intel 8259A PIC chip to manage hardware interrupts. A single 8259A can manage 8 interrupt lines (IRQ0-IRQ7), and through a master-slave cascading approach, it can be extended to 15 interrupt lines (IRQ0-IRQ15). The 8259A uses edge-triggered or level-triggered interrupt signals, with fixed priority (IRQ0 has the highest priority).

2.2 APIC (Advanced Programmable Interrupt Controller)

As the number of CPU cores increased, the traditional 8259A architecture encountered severe bottlenecks in multi-processor environments. The APIC architecture arose at the historic moment and mainly consists of two parts:

• Local APIC (LAPIC): Integrated into each CPU core, responsible for receiving and processing interrupts directed to the local CPU. LAPIC includes a timer, performance counters, and interrupt registers.
• I/O APIC: Usually integrated into the South Bridge chip, responsible for receiving interrupt signals from peripheral devices and distributing them to the appropriate LAPIC message interrupts. Modern I/O APICs typically support 24 interrupt lines (IRQ0-IRQ23).

2.3 MSI and MSI-X

MSI (Message Signaled Interrupts) is a modern interrupt mechanism introduced in PCI 2.2. Unlike traditional pin-based interrupts, MSI issues interrupt requests by writing specific data to a specific memory address hosted by the LAPIC. This approach avoids the sharing and competitive interrupt delay problems of physical IRQ lines and supports the simultaneous allocation of multiple interrupt vectors (MSI-X supports up to 2048 interrupt vectors).

2.4 Interrupt Affinity and Interrupt Balancing

In multi-core systems, interrupt affinity (Interrupt Affinity) determines which CPU cores a specific interrupt can be processed by. By setting /proc/irq/IRQ#/smp_affinity, users can bind specific interrupts to specific CPU cores to optimize cache locality and load balancing. irqbalance is an interrupt auto-balancing daemon that distributes interrupts to different CPU cores to avoid the lethality of a core becoming overloaded.

3. Interrupt Descriptor Table (IDT) and IDT Initialization

3.1 IDT Structure

The Interrupt Descriptor Table (IDT) is the basis for the x86 architecture to implement interrupt handling. It is essentially an array of 256 entries (on x86_64), with each entry being an 8-byte (32-bit mode) or 16-byte (64-bit mode) descriptor containing the following information:

• Handler function address (offset)
• Target code segment selector (segment selector)
• Gate type (Interrupt Gate / Trap Gate / Task Gate)
• Privilege level (DPL)
• Present bit (P)

The differences between Interrupt Gate and Trap Gate lie in interrupt masking behavior: Interrupt Gate will clear the IF flag to disable interrupts when entering the handler, preventing nested interrupts; Trap Gate does not modify the IF flag, allowing interrupts to be nested.

3.2 IDT Initialization Process

In the Linux kernel, IDT initialization is divided into two stages:

Stage 1: Early IDT Setup (setup_arch() → idt_setup_early_traps())

Initialized in the start_kernel() phase, this stage establishes a minimal IDT to handle critical exceptions (such as #DF Double Fault, #GP General Protection Exception, #PF Page Fault). At this point, memory management and scheduling systems are not yet running, and must use a statically allocated IDT.

Stage 2: Complete IDT Setup (idt_setup_traps())

Initialized after kernel startup is complete. It will:

1. Register a unified interrupt entry point for 256 vector numbers through the SETGATE() macro.
2. Register interrupt handling stubs for system calls (vector 0x80 on 32-bit, MSR_LSTAR on 64-bit syscall).
3. Register entries for local APIC interrupts (such as SPURIOUS_APIC_VECTOR, timer interrupt, etc.).

3.3 Interrupt Vector Number Allocation

On the x86_64 architecture, the Linux kernel's interrupt vector number allocation is as follows:

4. Complete Path of Hardware Interrupt Handling

4.1 Hardware Automatically Performed Actions

When the CPU receives an interrupt, it automatically executes the following operations (in hardware order):

1. Push the RFLAGS, CS, and RIP registers onto the current stack.
2. If the interrupt occurs at a different privilege level (e.g., from user mode to kernel mode), also push SS and ESP.
3. Clear the IF flag (if the interrupt uses an Interrupt Gate) to disable external maskable interrupts.
4. The target address is found based on the interrupt vector number and IDT entry.
5. Jump to the entry point of the interrupt handler.

4.2 Entry Point for the Interrupt Handling Stub

On x86_64, each interrupt vector corresponds to an assembly entry point registered by the interrupt macro in arch/x86/entry/entry_64.S. These entry points perform the following tasks:

1. Save the general-purpose registers (RAX, RCX, RDX, etc.) to the kernel stack.
2. For interrupts with error codes (e.g., #DF, #PF), process the error code.
3. Call the high-level C interrupt handler through the idtentry macro.

4.3 Call Flow of High-Level Interrupt Handling

The flow after entering high-level interrupt handling is as follows:

do_IRQ()
  → handle_irq_event()
    → handle_irq_event_percpu()
      → __handle_irq_event_percpu()
        → handler->action->handler()  // Call a series of interrupt action chains registered by the device driver

In do_IRQ(), the kernel first calls the processing function of the corresponding interrupt descriptor's irqchip (such as mask_ack_irq()) to mask and confirm the interrupt source, then traverses the handler list registered on that IRQ line, and calls each handler in sequence.

4.4 irqdesc and irqaction Data Structures

The interrupt descriptor struct irq_desc is the central data structure for the kernel to manage interrupts, containing:

• irq_common_data: Affinity, CPU mask, and parent hardware interrupt controller pointer.
• irq_data: Hardware interrupt number, domain (irq_domain), and hardware controller information.
• kstat_irqs: Per-CPU interrupt statistics.
• action: Pointer to the interrupt handler list (struct irqaction).
• depth: Interrupt disable nesting count.
• lock: Spinlock to protect concurrent access.

struct irqaction represents a registered interrupt handler which contains:

• handler: The interrupt handler function pointer (i.e., irq_handler_t).
• dev_id: Device identifier for shared interrupt lines.
• irq: Interrupt number.
• flags: Configuration flags such as IRQF_SHARED, IRQF_TRIGGER_RISING.

5. Top-Half and Bottom-Half: The Original Intent of Interrupt Splitting

5.1 Why Interrupt Splitting Was Needed

The original interrupt handling in the Linux kernel executed all processing directly in interrupt context. However, in practice, hardware interrupt handlers often require a large amount of data processing, such as network packet encapsulation parsing or filesystem metadata updates. Executing these time-consuming tasks in interrupt context leads to two serious problems:

1. Interrupt Loss: When an interrupt is being handled, the same interrupt is typically masked (masked). If the handler spends too long, the device issues a new interrupt (such as a network card receiving a new packet) during this period will be lost.
2. System Response Delay: Interrupt context disables the kernel's preemption capability (in some cases even interrupt nesting). If a large number of time-consuming operations are executed in interrupt context, the system's response time will significantly increase.

5.2 The Difference Between Top-Half and Bottom-Half

To solve this problem, Linux 2.3 introduced the interrupt handling splitting mechanism: interrupt handlers are split into a "Top-Half" (top half) that runs in interrupt context and a "Bottom-Half" (bottom half) that runs after the interrupt exits.

• Top-Half: The handler directly registered by the device driver via request_irq(). It executes in interrupt context (interrupt context/disabled interrupt), performing only time-critical tasks (such as reading hardware status registers, acknowledging interrupts, copying data to a kernel buffer). Execution time must be extremely short.
• Bottom-Half: Deferred execution mechanism, executes in a softer interrupt context or process-like context (depending on the specific mechanism), processing time-consuming tasks (such as protocol stack processing, file I/O, etc.). While the bottom half executes, it allows other interrupts to be responded to normally.

6. Softirq: A Kernel Low-Priority Interrupt Mechanism

6.1 Design Concept of Softirq

Softirq is one of the earliest preemptive (non-trappable) kernel mechanisms in the Linux kernel. Its core concept is: simulating a software-implemented interrupt to execute interrupt bottom-half tasks outside of hardware interrupt context. Softirq runs in a "soft interrupt" context, which allows hardware interrupts to be re-enabled while prohibiting the execution of higher-priority softirqs.

6.2 Priority and Types of Softirq

The Linux kernel defines 10 softirq types (since 5.10, the actual active types are less than 10), with the priority determined by the index in the enumeration enum list:

enum {
    HI_SOFTIRQ=0,          // High-priority softirq (tasklet_hi)
    TIMER_SOFTIRQ,         // Timer softirq
    NET_TX_SOFTIRQ,        // Network transmission
    NET_RX_SOFTIRQ,        // Network reception -- the most commonly used
    BLOCK_SOFTIRQ,         // Block device
    IRQ_POLL_SOFTIRQ,      // Interrupt polling
    TASKLET_SOFTIRQ,       // Tasklet (common)
    SCHED_SOFTIRQ,         // Scheduler
    HRTIMER_SOFTIRQ,       // High-resolution timer
    RCU_SOFTIRQ,           // RCU garbage collection -- usually the highest priority
};

6.3 Registration and Triggering of Softirq

Softirq usage follows a statically registered model:

1. Register a handler functions for the specified softirq_vec via open_softirq() (can only be registered at compile time or during module load, not dynamically).
2. Trigger the pending flag of the specified softirq by raising a __raise_softirq_irqoff(). Location is a CPU-specific unsigned integer irq_stat.__softirq_pending.

6.4 Execution Timing of Softirq

Softirq does not have an independent scheduling unit but is instead executed through the following trigger points:

• Hardware interrupt return path: irq_exit() in do_IRQ(). If there are pending softirqs, invoke invoke_softirq().
• ksoftirqd kernel thread: When the softirq trigger frequency is too high (if not processed within a certain time), the ksoftirqd/ kernel thread is awakened to execute softirq in SCHED context. This is to prevent starvation of the user processes caused by the time-consuming full occupation softirq.
• Explicit check points: Places where a context switch may occur in the kernel (such as cond_resched() when the kernel is preemptible).

6.5 Softirq Limitations

• Static Registration: Cannot be dynamically registered/removed, and must be defined through macros at compile time to ensure initialization.
• Strict Synchronization Requirements: Softirq handlers of the same type can execute synchronously on different CPUs, which means shared data must be protected strictly by per-CPU variables or lock mechanisms.
• Exclusive Execution: The same softirq type cannot execute simultaneously on different CPUs (by default, only when the handler explicitly invokes a rest of function triggers ksoftirqd can it be re-entrant).

7. Tasklet: A Bottom-Half Mechanism Built on Softirq

7.1 Design Goals of Tasklet

Tasklet is a higher-level mechanism built on top of softirq, designed to provide a simpler interface for device drivers to achieve bottom-half handling. Tasklet is internally implemented using two softirq numbers: TASKLET_SOFTIRQ (normal priority) and HI_SOFTIRQ (high priority).

7.2 Tasklet Data Structure

struct tasklet_struct {
    struct tasklet_struct *next;   // Pointer forming a linked list
    unsigned long state;           // TASKLET_STATE_SCHED / TASKLET_STATE_RUN
    atomic_t count;                // Disable counter
    void (*func)(unsigned long);   // Handler function
    unsigned long data;            // Parameters passed to the handler
};

Each CPU maintains two tasklet linked lists: tasklet_vec (normal) and tasklet_hi_vec (high priority). When tasklet_schedule() is called, the tasklet is inserted into the corresponding linked list and the softirq of TASKLET_SOFTIRQ is raised.

7.3 Tasklet Scheduling Rules

• A scheduled tasklet will execute only once on the same CPU.
• If the tasklet executes on one CPU, its TASKLET_STATE_SCHED location is signaled, even if tasklet_schedule() is called again before execution, it is guaranteed to be executed only once.
• Same tasklets cannot run simultaneously on different CPUs (compared to softirq, this is a stricter exclusivity convention).

7.4 Differences Between Tasklet and Softirq

8. Workqueue: Bottom-Half Execution in Process Context

8.1 Design Goals of Workqueue

Unlike softirq and tasklet executing in interrupt context, workqueue allows bottom-half tasks to execute in a specialized kernel thread (worker thread) in user-like (process context). The most significant advantage of this is: workqueue context allows blocking operations (mutex acquisition, memory I/O operations, sleep, etc.), so workqueue has much higher flexibility and complexity in handling long-running tasks.

8.2 Workqueue Architecture

The workqueue system is primarily organized into three layers:

1. API Layer: Provides schedule_work(), schedule_delayed_work(), and other interfaces.
2. cmwq (Concurrency Management Workqueue): Manages a pool of worker threads on each CPU and automatically adjusts the number of worker threads based on the system's load (Dynamic worker pool since Linux 5.10 may be replaced by a new bound worker pool).
3. Worker Thread (worker thread): kworker/n:k kernel threads in user-like context, executing actual work function functions from the pending queue.

8.3 Worker Thread Management Mechanism

8.4 How to Choose Between Workqueue and Tasklet/Softirq?

• If the bottom-half task needs to block/sleep: use workqueue.
• If the bottom-half task needs extremely high throughput and frequency (e.g., network packet capture) and does not block: use softirq or tasklet.
• If the bottom-half task has no special requirements: prioritize tasklet.

9. Threaded IRQ: A Modern Way of Interrupt Handling

9.1 Introduction to Threaded IRQ

Since Linux 3.1, kernel developers introduced request_threaded_irq() API, which allows device drivers to split interrupt handlers into two parts: hardirq handler (hard interrupt handler) and threaded handler (thread handler). The hardirq handler plays the role of top-half, executing in interrupt context, and quickly checking whether the interrupt originates from the device being driven; it returns IRQ_WAKE_THREAD, then the threaded handler executes in a dedicated kernel thread to complete complex business logic.

// Request a threaded interrupt
request_threaded_irq(irq, hardirq_handler, threaded_handler, flags, name, dev);

// hardirq_handler optionally returns IRQ_WAKE_THREAD to wake up the thread

9.2 Handling Flow of Threaded IRQ

1. An interrupt occurs, executes the hardirq handler (interrupts disabled).
2. If the hardirq handler returns IRQ_NONE, it indicates the interrupt is not from this device and the next handler continues.
3. If it returns IRQ_WAKE_THREAD, it wakes up the corresponding thread handler.
4. The thread executes thread_fn in SCHED context, which may block and sleep.
5. If the hardirq handler returns IRQ_HANDLED, processing is complete and the thread is not woken.

           CPU0       CPU1       CPU2       CPU3
  16:     12345          0          0          0   IO-APIC    16-fasteo   enp0s3
  17:          0      54321          0          0   IO-APIC    17-fasteo   enp0s8

                    CPU0       CPU1       CPU2       CPU3
      HI:          1234        567        890        234
   TIMER:         56789       3456       7890       1234
  NET_TX:           123        456        789        234
  NET_RX:       9876543     123456      78901      23456