XDP (eXpress Data Path): A Deep Engineering Analysis of In-Kernel High-Performance Packet Processing

1. Why XDP: The Performance Imperative

In the era of 100Gbps+ network interfaces, traditional Linux kernel networking stack processing has become a critical bottleneck. A single packet traversing the entire kernel network stack requires memory allocation, softirq scheduling, protocol layer processing, and finally reaches user space鈥攃onsuming approximately 1000-2000 CPU cycles in total. When traffic reaches 10 million packets per second, CPU resources are consumed by the data plane itself, leaving very little capacity for actual business logic.

XDP (eXpress Data Path) was born to solve this problem. It inserts an eBPF hook at the lowest layer of the NIC driver鈥攊mmediately after the DMA ring receives data packets and before the kernel allocates sk_buff, enabling data packet processing at the earliest point in the network stack. This design allows XDP programs to process each packet in approximately 100-200 cycles, more than 10 times faster than the traditional stack, achieving line-rate packet processing at 100Gbps.

The core advantages of XDP are reflected in the following dimensions: XDP programs execute in kernel space, avoiding user space-kernel context switches; zero-copy technology ensures data packets are always resident in DMA memory; hardware offload support allows XDP programs to run directly on NICs to achieve wire-speed processing; the eBPF verification mechanism ensures program safety and prevents kernel crashes.

2. XDP Architecture and eBPF Dependency Analysis

2.1 eBPF Subsystem Fundamentals

The execution of XDP programs relies on three core components of eBPF (Extended Berkeley Packet Filter): the verifier, the JIT compiler, and Maps. The verifier performs static analysis of bytecode to ensure program safety properties鈥攂ounded loops, no unreachable instructions, bounded memory access range, bounded stack depth, and correct helper function calls. The JIT compiler translates eBPF bytecode into native x86_64/ARM64 instructions, achieving execution efficiency close to native code. Maps is the shared state mechanism between eBPF programs and user space, supporting various data structures such as Hash, Array, Per-CPU Array, LRU Hash, LPM Trie, etc.

2.2 XDP Data Packet Lifecycle

Data packets arrive at the NIC DMA ring from the network cable, and the driver triggers NAPI polling. Before calling the kernel protocol stack, the XDP hook point checks the existence of an XDP program鈥攊f one exists, the program processes the packet; if the program returns XDP_PASS, the packet continues normally into the kernel protocol stack; if XDP_DROP is returned, the packet is immediately dropped; XDP_TX sends the packet back through the original NIC; XDP_REDIRECT sends the packet to another NIC or CPU.

3. XDP Program Development Full Process

3.1 eBPF C Program Writing

#define KBUILD_MODNAME "xdp_drop"
#include 
#include 
#include 
#include 

/* Define maps for counting dropped packets */
struct bpf_map_def SEC("maps") drop_counter = {
    .type = BPF_MAP_TYPE_PERCPU_ARRAY,
    .key_size = sizeof(u32),
    .value_size = sizeof(long),
    .max_entries = 1,
};

SEC("xdp")
int xdp_drop_prog(struct xdp_md *ctx)
{
    void *data_end = (void *)(long)ctx->data_end;
    void *data     = (void *)(long)ctx->data;
    struct ethhdr *eth = data;               /* Ethernet header */
    struct iphdr  *iph = data + sizeof(*eth); /* IP header */

    if (iph + 1 > (struct iphdr *)data_end)
        return XDP_PASS;

    /* Block TCP SYN packets - simple SYN flood protection */
    if (iph->protocol == IPPROTO_TCP) {
        struct tcphdr *tcph = (void*)iph + sizeof(*iph);
        if (tcph + 1 > (struct tcphdr *)data_end)
            return XDP_DROP;
        if (tcph->syn) {
            u32 key = 0; long *val;
            val = bpf_map_lookup_elem(&drop_counter, &key);
            if (val) __sync_fetch_and_add(val, 1);
            return XDP_DROP;
        }
    }
    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

3.2 Compilation and Loading Workflow

# 1. Compile C to eBPF object file
clang -O2 -g -target bpf -c xdp_drop.c -o xdp_drop.o

# 2. Load via iproute2 (native mode)
ip link set dev eth0 xdp obj xdp_drop.o sec xdp

# 3. Verify load status
ip link show eth0 | grep xdp

# 4. Read counters from maps
bpf map dump id 

# 5. Unload XDP program
ip link set dev eth0 xdp off

3.3 Userspace Management with libbpf and BPF CO-RE

For production firewalls and load balancers requiring dynamic configuration, we typically use libbpf combined with BPF CO-RE (Compile Once 鈥?Run Everywhere) to manage XDP programs. Libbpf handles ELF relocation, map creation, and kernel BTF matching automatically, enabling an XDP object file compiled on kernel A to load correctly onto a different version of kernel B. The CO-RE approach uses BTF (BPF Type Format) and Clang's __builtin_preserve_access_index to resolve kernel structure field offsets at load time, eliminating the need for recompilation for each target kernel version.

4. XDP Maps and Data Interaction

Maps are the core mechanism for data exchange between XDP programs and userspace, supporting various data structures optimized for different scenarios.

5. Production-Grade XDP Use Cases

5.1 DDoS Protection at Line Rate

Deploying XDP_DROP for SYN flood, UDP amplification, and DNS/NTP reflection attacks can block up to 100Mpps of attack traffic at the NIC driver layer. The following architecture combines LPM Trie for IP blacklisting and SYN cookies for legitimate connection validation. Production deployments show that a single 32-core server can handle 40Gbps of attack traffic while maintaining latency under 50渭s for legitimate flows.

/* DDoS mitigation XDP program - core logic */
SEC("xdp")
int xdp_ddos_filter(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end) return XDP_DROP;
    
    /* Only handle IPv4 */
    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return XDP_PASS;
    
    struct iphdr *iph = data + sizeof(*eth);
    if ((void *)(iph + 1) > data_end) return XDP_DROP;
    
    /* LPM Trie lookup for malicious IP prefixes */
    struct {
        __u32 prefix_len;
        __u32 addr;
    } key = { .prefix_len = 32, .addr = iph->saddr };
    
    __u8 *action = bpf_map_lookup_elem(&ip_blacklist, &key);
    if (action && *action == BLACKLIST_DROP)
        return XDP_DROP;
    
    /* UDP amplification attack detection */
    if (iph->protocol == IPPROTO_UDP) {
        struct udphdr *udph = (void*)iph + sizeof(*iph);
        if ((void *)(udph + 1) > data_end) return XDP_DROP;
        /* Drop UDP port 19 (chargen), 123 (NTP), 1900 (SSDP) for untrusted sources */
        __u16 dport = bpf_ntohs(udph->dest);
        if (dport == 19 || dport == 1900)
            return XDP_DROP;
        /* NTP monlist attack - flag 0x17 with mode 7 */
        if (dport == 123 && udph->len > bpf_htons(256))
            return XDP_DROP;
    }
    
    /* SYN rate limiting per source IP */
    if (iph->protocol == IPPROTO_TCP) {
        struct tcphdr *tcph = (void*)iph + sizeof(*iph);
        if ((void *)(tcph + 1) > data_end) return XDP_DROP;
        if (tcph->syn && !tcph->ack) {
            __u64 *pkt_cnt = bpf_map_lookup_elem(&syn_count, &iph->saddr);
            if (pkt_cnt && *pkt_cnt > SYN_THRESHOLD)
                return XDP_DROP;
        }
    }
    
    return XDP_PASS;
}

5.2 Layer-4 Load Balancer

Facebook's Katran project is a masterpiece of XDP-based Layer-4 load balancing. Its core algorithm stores backend server addresses and connection mapping tables (stable mode) in Hash maps. When a request packet arrives, the XDP program looks up the existing connection mapping or assigns a new backend server based on CPU core ID (consistent hashing), then rewrites destination MAC/IP/Port and redirects the packet to the target backend via XDP_REDIRECT. This architecture achieves 10x throughput improvement over IPVS, supporting millions of connections per second with p99 latency under 200渭s.

5.3 eXpress Data Path to Userspace (AF_XDP)

AF_XDP allows redirecting packets directly to userspace applications via XDP_REDIRECT to an XSKMAP, bypassing the kernel network stack entirely. This zero-copy path maps NIC ring buffers into userspace via mmap, achieving throughput close to DPDK while retaining kernel protocol stack capabilities. Use cases include: custom protocol parsing, intrusion detection systems (Suricata), and application-layer load balancing.

# AF_XDP benchmark setup
# 1. Create UMEM area
./xdpsock --ifname eth0 --rx --zero-copy

# 2. Performance comparison (single core, 64B packets)
#    Traditional kernel: ~1.2Mpps
#    AF_XDP (zero-copy): ~18Mpps
#    DPDK (zero-copy):   ~20Mpps

# 3. CPU efficiency at 10Gbps line rate
#    Kernel stack:  100% of one core
#    AF_XDP:        ~35% of one core
#    DPDK:          ~32% of one core

6. XDP Advanced Techniques

6.1 Tail Calls (BPF-to-BPF Calls)

Although XDP programs are limited to a single function, tail calls (bpf_tail_call) allow linking multiple XDP programs into a processing pipeline. The tail call updates the CPU's RSP register and jumps to the target program without a function call overhead, enabling modular design鈥攆or example, separating parsing, detection, and forwarding into independent XDP programs for individual development and testing.

/* Tail call program jump table */
struct bpf_map_def SEC("maps") xdp_progs = {
    .type = BPF_MAP_TYPE_PROG_ARRAY,
    .key_size = sizeof(u32),
    .value_size = sizeof(u32),
    .max_entries = 4,
};

SEC("xdp")
int xdp_entry(struct xdp_md *ctx) {
    bpf_tail_call(ctx, &xdp_progs, XDP_STAGE_PARSE);
    return XDP_PASS; /* fallback */
}

SEC("xdp")
int xdp_parse(struct xdp_md *ctx) {
    /* Parse packet headers, extract 5-tuple */
    bpf_tail_call(ctx, &xdp_progs, XDP_STAGE_FILTER);
    return XDP_PASS;
}

SEC("xdp")
int xdp_filter(struct xdp_md *ctx) {
    /* Apply ACL rules, rate limiting */
    bpf_tail_call(ctx, &xdp_progs, XDP_STAGE_FORWARD);
    return XDP_PASS;
}

SEC("xdp")
int xdp_forward(struct xdp_md *ctx) {
    /* Rewrite L2/L3 headers, redirect */
    return XDP_REDIRECT;
}

6.2 Hardware Offloading

Some SmartNICs (NVIDIA ConnectX-4/5/6, Netronome Agilio) and certain driver modes (e.g., mlx5 for ConnectX-5) support hardware offloading of XDP programs. In this mode, the eBPF program is translated directly into NIC firmware instructions and runs entirely on the NIC processor, consuming no host CPU at all. Notable limitations: all maps must be NIC-local, helper functions are restricted to a subset, and debugging requires vendor tools. Production offloading rates can reach 200Mpps with zero host CPU utilization.

6.3 XDP Binding to Single Queues

In multi-queue NIC environments, XDP programs can be bound to individual queues for precisely controlling CPU affinity. Combined with CPUMAP redirect, a hierarchical architecture is achieved: the first XDP performs preliminary classification, and the second XDP (bound to the target queue) performs deep inspection and forwarding. This design resembles a programmable pipeline with stages running on different CPU cores.

7. XDP Performance Analysis and Best Practices

7.1 Performance Tuning Checklist

• Enable JIT: sysctl net.core.bpf_jit_enable=2 (mode 2 enables debug prints to dmesg)
• Optimize map memory: Pre-allocate maps with exact size at startup to avoid runtime hash table rehash
• Batch operations: Use bpf_map_lookup_elem_per_cpu for reduced cache line bouncing in per-CPU maps
• Avoid expensive helpers: bpf_skb_store_bytes and bpf_csum_level are expensive in XDP context
• Instruction count: Keep under 4096 instructions for maximum portability across drivers
• CPU affinity: Pin NAPI polling and XDP processing to the same core using irqbalance and IRQ affinity

8. Debugging and Observability

# 1. Check XDP program load status
ip -d link show eth0
bpftool prog show

# 2. View JIT-compiled code
bpftool prog dump xlated id 
bpftool prog dump jited id 

# 3. Trace XDP return codes
bpftrace -e 'tracepoint:xdp:xdp_exception { @[args->action] = count(); }'

# 4. Performance stats
bpftool prog show --json | jq '.[] | select(.run_time_ns > 0)'

# 5. XDP test environment
#    Use veth pair for local XDP testing without real NIC
ip link add dev veth0 type veth peer name veth1
ip link set dev veth0 xdp obj xdp_drop.o

9. Limitations and Trade-offs

• sk_buff unavailable: XDP programs only access raw packets (ctx->data / ctx->data_end), cannot access socket buffers, so conntrack, IP fragments reassembly, and other kernel network stack features are unavailable. Use Cases requiring this functionality must be processed in TC BPF or user mode after XDP_PASS.
• No loops: The Verifier forbids loops and ensures that the XDP Verifier runs in polynomial time. Any potential unbounded loop merge must be unrolled. Complex algorithms such as L7 protocol analysis are difficult to implement directly in XDP and should be handled using tail calls or AF_XDP postscript bypass.
• Memory limitations: The eBPF stack is only 512 bytes, and complex packet processing must store data using map prefetch rather than the stack. Cloning large packets is impossible; XDP can only trim packets by returning modified pointers.
• Driver support gaps: Not all NIC drivers support XDP; some require Generic XDP (gXDP), which hangs on NAPI context but incurs sk_buff creation overhead.

10. Summary

XDP represents the most significant evolution in Linux kernel networking since the introduction of the traffic control (tc) framework. It achieves wire-rate packet processing at 100Gbps+ while maintaining full compatibility with the existing kernel stack. The trio of hardware offloading, tail calls, and AF_XDP extends XDP's applicability from simple DDoS filtering to high-performance load balancing, user-space gateways, and programmable switch pipelines. As eBPF ecosystem tools (bcc, bpftool, Cilium Hubble) mature, XDP is becoming an essential technology for modern cloud-native infrastructure.

For engineers, mastering XDP means: understanding eBPF constraints, ICMP/DHCP/layer 2 control protocols bypass, exploiting CO-RE portability, and designing tail call pipelines that separate parsing and forwarding. The 10 lines of C code at the beginning of this article and the production-grade firewall cases at the back form a proving ground where the kernel can accommodate packets at network speed.