Test After 13k

ical paths. This breaks down as: 2s detection window + 1s decision + 2s action execution + <1s validation. Production teams that achieve this threshold report a reduction in user-visible downtime from 12 minutes (MTTR) to near-zero.

7. Resilience Testing: Chaos Engineering for LLM Systems

A resilience architecture that has never been tested is not resilient - it's wishful thinking. Chaos engineering adapted for AI systems injects failures into production (or staging) to validate mitigation paths.

7.1 LLM-Specific Chaos Experiments

• PROVIDER_KILL: Simulate full provider outage by blocking egress traffic to one provider's IPs. Verify automatic multi-model routing activates within detection SLA.
• LATENCY_INJECTION: Add 5-15s artificial delay to model responses. Verify circuit breaker trips and p99 SLO holds via fallback path.
• RATE_LIMIT_STORM: Blast the system with traffic that exceeds token quota. Verify graceful queue prioritization and degradation triggers.
• SEMANTIC_DEGRADATION: Inject prompt templates known to cause hallucination in the target model. Verify semantic circuit breaker trips before users see degraded outputs.
• CASCADING_AGENT_FAILURE: Kill a downstream tool API mid-agent-loop. Verify the agent detects tool failure, reports clearly, and does not enter retry storm or context bloat.

7.2 GameDay Protocols

Run at least one LLM GameDay per sprint with the following structure:

1. Steady State Baseline: 30 minutes of normal traffic to establish p99 latency, error rate, and user-satisfaction signal.
2. Failure Injection: Execute pre-defined experiment from the chaos catalog above.
3. Observation: Measure detection time, mitigation time, user-visible impact duration, and automatic recovery time.
4. <stro