Chaos engineering has a reputation for being about breaking things for fun. The practice is closer to a lab experiment. Instead of describing it in the abstract, let us run one, start to finish, on an invented system: the checkout service of a fictional food-delivery company, Dosa Dash.
The system and the worry
Checkout reads the restaurant’s menu and prices from a cache (Redis), falling back to the main database when the cache has no answer. The architecture diagram says the cache is “optional, for speed”. Nobody has ever tested what happens when it vanishes. That unverified belief is the worry, and it is what we will convert into a testable statement.
Step 1: define the steady state
Before breaking anything, decide what “healthy” looks like in numbers the business cares about. For Dosa Dash: checkout success rate above 99%, p95 checkout latency under 1.5 seconds, and error rate on the order API under 0.5%. These are read from the dashboards, in production or a production-like environment, before and during the experiment. Without a baseline you cannot say anything changed.
Step 2: write the hypothesis
“If one of the three cache nodes is terminated, the steady-state metrics remain within limits, because requests will fail over to the other nodes or fall back to the database.” Notice it is falsifiable, and it names the reason. If the experiment disproves it, we learn something specific.
Step 3: limit the blast radius
Start in a staging environment that carries realistic synthetic load. Only after it passes there, repeat in production during a low-traffic window, affecting a small share of traffic (for instance a single availability zone), with the on-call engineer present. Agree the abort condition before starting: if checkout success drops below 97% for two consecutive minutes, stop and restore. A one-line “stop button” that someone can press in seconds is part of the design, not an afterthought.
Step 4: inject the fault
Terminate one cache node. Tools help with this: Chaos Monkey from Netflix, which randomly terminates instances, was the early famous one; others include Gremlin, Chaos Mesh and LitmusChaos for Kubernetes, and AWS Fault Injection Service for AWS workloads. You can also do it by hand with an ordinary command. The tool matters less than the discipline.
Step 5: observe
Suppose the result in staging is this. For the first 40 seconds, p95 latency climbs from 0.9 to 4 seconds and the error rate hits 6%. Then it recovers. The logs reveal why: the cache client was configured with a 5-second timeout and no retry on another node, so requests that hashed to the dead node waited five seconds and then failed over to the database. The database, suddenly receiving every menu lookup that missed, ran hot for a while.
The hypothesis was wrong, and this is the success case. A defect that would have appeared at 8 p.m. on a Saturday with real customers appeared on a Tuesday afternoon in staging, with everyone calm and watching.
Step 6: fix, then rerun
The team lowers the cache timeout to 200 milliseconds, adds a retry on a second node, adds a circuit breaker so a failed node is skipped quickly, and puts a short-lived in-process cache of popular menus in front of the database to absorb the surge. They rerun the identical experiment. This time p95 rises from 0.9 to 1.2 seconds for a few seconds and errors stay under 0.5%. Hypothesis now supported. The experiment is added to a recurring schedule, because systems change and yesterday’s resilience does not last.
What this teaches beyond the example
- It tests assumptions in the architecture, mostly about timeouts, retries and fallbacks, rather than the correctness of features.
- Good experiments are small and boring: one fault, one hypothesis, one clear measure. Killing half the data centre on day one teaches little.
- It needs monitoring that works. If your dashboards cannot show success rate by the minute, fix that before any fault injection.
- Other common faults: added network latency, packet loss, a dependency returning errors, a full disk, a clock jump, a dependency that is slow rather than down. Slow is usually nastier than dead, because timeouts, not errors, decide the outcome.
Not every organisation is ready. If you have no on-call rota, no rollback, and no agreed targets, start with plain failure drills in a test environment: restart a service, cut a network link, see what the logs say. Production experiments are the late stage, and for regulated systems should be cleared with risk and compliance teams beforehand.
Figures and names in the examples are illustrative.
Leave a Reply