2.2 v2 Unified Hierarchy Design

Tejun Heo's redesign enforced single unified hierarchy with three cardinal rules:

The single-hierarchy constraint means a process's resource configuration is unambiguous: /sys/fs/cgroup/system.slice/nginx.service applies all controllers simultaneously to that path.

3. Kernel Internals: The cgroup Subsystem

3.1 Data Structures

At the kernel level, cgroups v2 operates through core data structures defined in linux/cgroup-defs.h:

struct cgroup_subsys_state {
    struct cgroup *cgroup;           /* associated cgroup */
    struct percpu_ref refcnt;       /* reference count */
    struct cgroup_subsys *ss;       /* subsystem pointer */
    struct rcu_work destroy_rcu;    /* RCU deferred destruction */
    /* Controller-specific inline data follows */
};

struct cgroup {
    struct cgroup_subsys_state self;  /* generic state */
    unsigned long flags;              /* CGRP_* flags */
    int level;                         /* depth from root */
    int max_depth;                     /* maximum allowed depth */
    int nr_descendants;                /* count of descendants */
    int nr_dying_descendants;          /* zombie count */
    struct cgroup *parent;             /* parent pointer */
    struct kernfs_node *kn;            /* sysfs/kernfs node */
    /* Resource distribution model */
    struct cgroup_ppd *root_ppd;       /* per-parameters for resource model */
};

3.2 The Resource Distribution Model

The v2 "Resource Distribution Model" solved v1's "no safe defaults" problem. Each non-leaf cgroup configures resource sharing between its children via three parameters:

The key conceptual advance is the memory tiered protection model: memory.min (reservation) → memory.low (best-effort) → memory.high (throttle) → memory.max (kill). This provides a graceful degradation curve rather than the binary "limit-or-OOM" paradigm of v1.

3.3 The no Internal Process (thread_domain) Rule

v2 enforces that resource domain controllers (cpu, io, memory) and threaded controllers cannot both be enabled on a non-leaf cgroup. This is expressed through cgroup.controls and cgroup.subtree_control interaction:

$ cat /sys/fs/cgroup/cgroup.controllers
cpu io memory pids

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory

$ echo '+cpu +memory -io' > /sys/fs/cgroup/webserver/cgroup.subtree_control
$ cat /sys/fs/cgroup/webserver/cgroup.subtree_control
cpu memory

4. Controller Deep Dives

4.1 Memory Controller — Stateless but Powerful

The memory controller is frequently the most critical for production workloads. v2's memory controller is stateless by default — it does not maintain consumption accounting unless explicitly requested. The statistics file memory.stat provides:

// memory.stat — comprehensive per-cgroup accounting
anon 16777216        // Anonymous pages (heap, mmap(MAP_ANON))
file 4194304         // Page cache (file-backed mmap)
kernel_stack 131072  // Kernel stack pages
pagetable 8192       // Page table slab
percpu 2048          // Per-CPU allocations
sock 0               // Network socket buffers
shmem 1048576        // Shared memory segments
file_mapped 2097152  // File-mapped pages
file_dirty 524288    // Dirty pages awaiting writeback
file_writeback 0     // Pages under active writeback
inactive_anon 4096   // COLD anonymous pages
active_anon 16773120 // HOT anonymous pages
inactive_file 1024   // COLD file cache
active_file 4184064  // HOT file cache
unevictable 0        // Pages pinned (mlock, zram)

The memory.current exposes total consumption; memory.peak (added in kernel 5.19) captures the historical maximum — invaluable for capacity planning.

4.2 CPU Controller — Weighted Fair Queuing

The v2 CPU controller replaced v1's cpu.shares with a normalized weight semantic. Unlike v1 where shares was relative only when there was contention, v2's cpu.weight operates within a scheduling period (default 100ms, configurable via cpu.cfs_period_us):

Process A: weight=100 → receives ~50% CPU when A+B contend 1:1
Process B: weight=100 → receives ~50% CPU when A+B contend 1:1
Process C: weight=300 → receives ~75% CPU when A+B+C contend 1:1:3

The controller supports two throttling modes via cpu.max (format $MAX $PERIOD):

# Guarantee 1 CPU worth of time per 100ms period
echo "100000 100000" > /sys/fs/cgroup/container/cpu.max

# Burst mode: allow up to 2 CPUs usage, but bounded over 100ms
echo "200000 100000" > /sys/fs/cgroup/container/cpu.max

# Read current usage metrics
cat /sys/fs/cgroup/container/cpu.stat
# usage_usec 847392847
# user_usec 612839284
# system_usec 234553563
# nr_periods 18472           // scheduling periods elapsed
# nr_throttled 892           // periods where cgroup was throttled
# throttled_usec 447392847   // total wall-clock throttled time

4.3 IO Controller — From Throttling to Latency Targeting

The v2 IO controller (io, kernel 5.0+) represented a philosophical shift from v1's bandwidth-centric model to a latency-first design:

• io.latency: Set a target latency (in microseconds) per device; kernel has deployed a feedback-loop mechanism that automatically adjusts throttling to meet SLOs.
• io.weight: Weighted proportional sharing (replaces v1's blkio.weight with normalized range 1–10,000).
• io.max: Hard per-device throttling in bytes/sec and IOPS.
• io.stat: Detailed per-device request latency histograms.

# Ensure device 8:0 (sda) maintains ≤ 5ms read latency
echo "8:0 target=5000" > /sys/fs/cgroup/db/cgroup/io.latency

# Throttle backup-container to 50MB/s write and 5000 IOPS on sda
echo "8:0 wbps=52428800 wiops=5000" > /sys/fs/cgroup/backup-container/io.max

# Inspect actual latency histogram
cat /sys/fs/cgroup/db/io.stat
# 8:0 rbytes=10737418240 wbytes=2147483648rios=2621440 wios=524288
# 8:0 dbytes=0 dios=0
# 8:0 ravg=2345 wavg=6789 cost=1234 cost.pr=2000 cost.max=8000

4.4 Pids Controller — Fork Bomb Protection

The pids controller is the simplest and oldest controller, protecting against fork bombs:

# Limit a cgroup subtree to 512 total processes
echo "512" > /sys/fs/cgroup/sandbox/pids.max

# Monitor in real-time
cat /sys/fs/cgroup/sandbox/pids.current  # current process count
cat /sys/fs/cgroup/sandbox/pids.peak     # historical peak (kernel 5.19+)
cat /sys/fs/cgroup/sandbox/pids.events   # max: 0 (or count if limit hit)

5. cgroup v2 Controllers with eBPF — Programmable Resource Management

5.1 cgroup-BPF Program Types and Attachment Modes

cgroup v2 extended the BPF subsystem with dedicated program types for resource control and tracing. These attach to cgroup filesystem nodes via BPF multi-link (BPF_F_MULTI) since kernel 5.7:

5.2 Case: Per-Service Socket Redirect with cgroup-BPF

A production pattern: transparently redirect all outbound connections from a container's cgroup to a local proxy, without modifying application code:

// cgroup_sock_addr.bpf.c — redirects :80 to :8080 transparently
SEC("cgroup/connect4")
int cgroup_connect4(struct bpf_sock_addr *ctx)
{
    // Only intercept IPv4 TCP connections to port 80
    if (ctx->user_family != AF_INET || ctx->protocol != IPPROTO_TCP)
        return 1;
    if (ctx->user_port != bpf_htons(80))
        return 1;

    // Modify destination to localhost:8080
    ctx->user_ip4 = bpf_htonl(0x7F000001);  // 127.0.0.1
    ctx->user_port = bpf_htons(8080);
    return 1;  // 1 = allow (with modification)
}

The loader attaches this program to /sys/fs/cgroup/web-proxy/cgroup/bpf/connect4, affecting every process in web-proxy without modifying any service code.

6. Container Runtimes: Docker, containerd, and Kubernetes Integration

6.1 Docker and cgroups v2

Docker uses cgroups (v2 when available) for all resource constraints. Key mapping examples from Docker CLI to cgroups files:

# Inspect running container's cgroup (requires cgroupv2 support)
$ docker run -d --name nginx --cpus=0.5 --memory=256m nginx

# Find the container's cgroup path
$ docker inspect --format '{{.State.Pid}}' nginx
12345

$ cat /proc/12345/cgroup
0::/system.slice/docker-abc123.scope

# Inspect applied limits
$ cat /sys/fs/cgroup/system.slice/docker-abc123.scope/cpu.max
50000 100000
$ cat /sys/fs/cgroup/system.slice/docker-abc123.scope/memory.max
268435456

6.2 Kubernetes Pod Resource Model on cgroups v2

Kubernetes maps Pod QoS classes directly to cgroup v2 resource knobs:

Starting with Kubernetes 1.28 (stable), the InPlacePodVerticalScaling feature (alpha) uses the cgroup v2 memory.low → memory.max reconciliation to perform live container resize without pod restart.

7. Production Tuning: Pressure Stall Information (PSI)

7.1 The PSI Tracepoint

Kernel 4.20 introduced Pressure Stall Information (PSI), the first hardware-agnostic, feedback-driven resource monitoring framework. PSI tracks task stalls into three resource dimensions: CPU, memory, and IO.

The format of memory.pressure, cpu.pressure, and io.pressure:

$ cat /sys/fs/cgroup/webserver/memory.pressure
some avg10=0.00 avg60=12.45 avg300=45.67 total=847392847
# "some" = time when at least ONE task is stalled on memory
# avg10/avg60/avg300 = % of last 10s/60s/300s at least one task stalled
# total = cumulative microseconds since cgroup creation

$ cat /sys/fs/cgroup/webserver/memory.pressure
full avg10=0.00 avg60=5.23 avg300=23.45 total=428392847
# "full" = time when ALL non-idle tasks are stalled simultaneously

7.2 PSI-Driven Autoscaling Architecture

Modern container orchestrators implement reactive hysteresis loops based on PSI metrics:

# pseudocode: PSI-triggered memory expansion
def evaluate_memory_pressure(cgroup_path):
    with open(f"{cgroup_path}/memory.pressure") as f:
        pressure_stats = parse_psi(f.read())
    
    some_avg60 = pressure_stats['some']['avg60']  # % of last 60s at least one task stalled
    
    if some_avg60 > 60:  # >60% of the time, tasks are memory-stalled
        current_max = read_memory_max(cgroup_path)
        new_max = min(current_max * 1.25, HARD_CEILING)
        write_memory_max(cgroup_path, new_max)
        cgroup_events.emit("memory.pressure.scale_up", factor=1.25)
    
    elif some_avg60 < 10 xss=removed xss=removed factor=0.83)>

$ cat /etc/oomd/oomd.conf
[OOM]
DefaultMemoryPressureDurationSec=30  # react only if pressure sustains 30s
MemoryPressureLimit=60                # trigger at 60% "some" pressure over 30s
DefaultMemoryPressureLimit=60%        # per-cgroup default

[Slice]
# System-level protection
MemoryMin=1G           # Never reclaim below 1GB for system.slice
StartupIOWeight=100
IOWeight=100

# User protection (ssh logins)
user.slice.MemoryHigh=4G

# Move a thread to a different cgroup while keeping the process elsewhere
$ echo 12345 > /sys/fs/cgroup/high-priority/cgroup.threads

# List all thread IDs in a cgroup
$ cat /sys/fs/cgroup/worker/cgroup.threads
12345
12346
12347

# Convert a resource domain controller to threaded (allow threads in subtree)
$ echo "threaded" > /sys/fs/cgroup/worker/cpu.threaded

# Check current cgroup version
$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)  # ✅ v2 only

# Force v2-only boot (kernel cmdline)
# In /etc/default/grub:
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"
# Then: sudo update-grub && reboot

# After reboot: verify
$ stat -fc %T /sys/fs/cgroup
cgroup2fs
$ cat /sys/fs/cgroup/cgroup.controllers
cpu io memory hugetlb pids rdma misc

$ systemd-cgls —list
Control group /:
-.slice
├─system.slice
│ ├─nginx.service
│ │ ├─12345 "nginx: master process /usr/sbin/nginx"
│ │ ├─12346 "nginx: worker process"
│ │ └─12347 "nginx: worker process"
│ └─docker-abc123.scope
│   ├─128845 /bin/sh
│   └─129552 node server.js
└─user.slice
  └─user-1000.slice
    └─session-3.scope
      └─15432 bash

# Deep drill on a specific unit with resource summary
$ systemd-cgtop -m -n 3
Control Group                     Tasks   %CPU   Memory  I/O/s Input/s Output/s
/system.slice/docker-abc123.scope   4    45.2   256.0M   12.4K    1.2K   8.7K
/system.slice/nginx.service          3     8.3    48.0M    2.1K  145.0  890.0
/system.slice/mysql.service         12   78.1     4.2G  234.5K   15.6K  34.2K

# Monitor all cgroup migrations in real-time
$ sudo bpftrace -e 'tracepoint:cgroup:cgroup_attach_task { 
    printf("task %s migrated to cgroup %s by PID %d\n", 
           args->task->comm, args->cgrp->kn->name, pid); 
}'

# Track OOM events (not just kills)
$ sudo bpftrace -e 'kprobe:oom_kill_process { 
    printf("OOM: killed PID %d (%s), cgroup %s, totalpages=%lu\n", 
           pid, comm, "try_cgroup_path(args->memcgp)", args->totalpages); 
}'

# Count cgroup write operations by file name (detect config churn)
$ sudo bpftrace -e 'kprobe:cgroup_file_write { 
    @[str(args->path->filename)] = count(); 
}'
^C
@memory.max: 347
@cpu.max: 128
@cgroup.procs: 89

# Cilium Tetragon TracingPolicy scoped by cgroup
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: cgroup-socket-audit
spec:
  kprobes:
  - call: "sock_alloc"
    selectors:
    - matchBinaries:
      - "/usr/bin/curl"
      matchArgs:
      - index: 0
        operator: "Postfix"
        values:
        - "cgroup/web-frontend/*"
    syscall: false

# libvirt/QEMU automatically creates:
/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2dwebserver.scope/

# Apply IO limits to the VM (vhost-user front-end)
echo "8:0 rbps=104857600 wbps=52428800" \
  > /sys/fs/cgroup/machine.slice/machine-qemu*webserver.scope/memory.max

memory.min hierarchy (cgroup v2 best practice):
/sys/fs/cgroup/            ← root: no min (everything passes through)
├── system.slice           ← memory.min = 1G (kernel + systemd services reserved)
├── user.slice             ← memory.min = 512M (user sessions reserved)
└── machine.slice          ← memory.min = 0 (VMs get what VMs get)

A well-behaved system under memory pressure:
1. Reclaim from machine.slice (VM guests first)
2. Then user.slice discretionary cache
3. Finally system.slice page cache (but not memory.min)

#!/bin/bash
# setup-webserver-cgroup.sh — production-ready cgroup v2 bootstrap
# Usage: ./setup-webserver-cgroup.sh webserver 50 256 1000

CGROUP_NAME=$1
CPU_WEIGHT=$2         # 1-10000
MEMORY_MB=$3
PIDS_MAX=$4
CGROUP_V2="/sys/fs/cgroup/${CGROUP_NAME}"

mkdir -p "$CGROUP_V2"

# Enable controllers on parent
echo "+cpu +memory +io +pids" > /sys/fs/cgroup/cgroup.subtree_control

# CPU: weight proportional to siblings, burst to 2 CPUs
echo "200000 100000" > "$CGROUP_V2/cpu.max"
echo "$CPU_WEIGHT" > "$CGROUP_V2/cpu.weight"

# Memory: hard limit + 20% throttle headroom
echo "$(( MEMORY_MB * 1024 * 1024 ))" > "$CGROUP_V2/memory.max"
echo "$(( MEMORY_MB * 1024 * 1024 * 8 / 10 ))" > "$CGROUP_V2/memory.high"
echo "$(( MEMORY_MB * 1024 * 1024 * 5 / 10 ))" > "$CGROUP_V2/memory.low"

# IO: 10MB/s burst cap per NVMe; 5ms latency target
echo "8:0 rbps=10485760 wbps=10485760" > "$CGROUP_V2/io.max"
echo "8:0 target=5000" > "$CGROUP_V2/io.latency"

# Pids: hard process count limit
echo "$PIDS_MAX" > "$CGROUP_V2/pids.max"

# Optional: freeze support for live migration
echo 1 > "$CGROUP_V2/cgroup.freeze"   # test freeze workflow

echo "Cgroup ${CGROUP_NAME} provisioned:"
echo "  cpu.weight     = $(cat $CGROUP_V2/cpu.weight)"
echo "  memory.max     = $(cat $CGROUP_V2/memory.max | numfmt --to=iec) bytes"
echo "  pids.max       = $(cat $CGROUP_V2/pids.max)"
echo "  subtree_ctrl   = $(cat $CGROUP_V2/cgroup.subtree_control)"