The incident timeline reads oddly. An inventory service goes down for five minutes after a bad configuration push. The fix is rolled out, the service starts, health checks pass, and within thirty seconds it falls over again. Restart, pass, fall. The pattern repeats four times before someone notices that the service is not broken any more. It is being crushed by the traffic that piled up while it was away. This is a retry storm, sometimes called a thundering herd, and it is one of the most common ways that a short outage becomes a long one.

What piles up while a service is down

A healthy system has some steady arrival rate, say 2,000 requests per second, and the inventory service has been sized to handle perhaps 3,000. During the outage, demand does not pause. Several things accumulate at once:

  • Client retries. Every caller that got an error is retrying, often on a fixed schedule. Users are also pressing refresh.
  • Queued work. Messages destined for the service sit in a broker. When consumers reconnect, they try to drain five minutes of backlog as fast as they can.
  • Cold state. The service’s caches are empty and its connection pools are not yet filled, so each request costs more than usual for the first minutes.
  • Reduced capacity. If an autoscaler saw low CPU while the service was failing fast, it may have scaled down. The service comes back with fewer instances than it needs.

So at the moment of recovery, the service faces several times its normal load while being less able to handle it. It slows, timeouts fire, callers retry, and the load increases further. The health check, which typically tests only that the process is alive, passes the whole time.

Retry amplification across layers

Retries multiply across a call chain. Imagine a mobile app calls an API gateway, which calls an order service, which calls inventory. If each of the three layers makes up to 4 attempts (one try plus three retries) when its downstream fails, a single user action can produce 4 × 4 × 4 = 64 attempts against the inventory service. Each layer’s retry policy looked reasonable on its own. Together they turn a struggling dependency into a target.

A widely followed guideline is to retry at one layer only, usually closest to the failing dependency or at the edge, and have other layers pass failures through. Where that is not possible, retry budgets (below) cap the damage.

Backoff and jitter

The first fix is to stop retrying at a fixed interval. With exponential backoff, the wait doubles after each failure: perhaps 100 ms, 200 ms, 400 ms, 800 ms, up to a cap. Clients that keep failing slow down on their own, giving the dependency room.

Backoff alone has a flaw. If 10,000 clients all failed at the same instant, they all retry after 100 ms, then all again at 300 ms, in synchronised waves. Jitter adds randomness to each wait so retries spread across the interval instead of arriving in spikes. The AWS Architecture Blog’s well-known analysis of backoff and jitter compared variants and found that “full jitter”, choosing a random wait between zero and the current backoff ceiling, did well at reducing both total work and contention. The exact variant matters less than having some randomness.

Also give up eventually. A retry that succeeds after 40 seconds is useless if the user left after 5. Set a maximum number of attempts and an overall deadline, and make sure the operation is safe to retry. Retrying a non-idempotent request, such as a payment without an idempotency key, can do real harm.

Budgets and breakers

A retry budget limits retries as a proportion of normal traffic. A client might allow retries to add at most 10% on top of its first attempts. When the dependency is mostly healthy, the occasional failure is retried. When it is badly unhealthy, the budget is quickly exhausted and the client stops adding load. gRPC’s retry throttling works on a related idea, using a token bucket that drains on failures and refills on successes.

A circuit breaker goes further. After the error rate crosses a threshold, it “opens” and fails calls immediately for a cooling period, without contacting the dependency at all. Then it lets a small number of trial requests through (the half-open state) and closes only if they succeed. Breakers protect the caller’s own threads from waiting on a dead service and protect the recovering service from a full flood. Libraries such as Resilience4j for Java and Polly for .NET implement this pattern, and service meshes such as Istio offer related outlier detection.

Making the recovering side robust

Client discipline helps, but you cannot control every client, especially mobile apps already in users’ hands. The recovering service needs its own defences.

Load shedding. When concurrent requests exceed what the service can process within its deadline, reject the excess immediately with a clear “try later” status such as 503 or 429, ideally with a retry-after hint. Rejecting quickly is cheap. Accepting work you cannot finish in time is expensive, because it consumes resources and then times out anyway, wasting everything spent on it.

Slow start. Several load balancers, including Envoy and AWS Application Load Balancer, support a slow-start mode in which a newly healthy target receives a gradually increasing share of traffic. Instances get time to warm caches and pools before taking a full load.

Controlled backlog drain. Queue consumers should process at a bounded rate after an outage, not at maximum speed. Draining five minutes of backlog over twenty minutes is usually fine for asynchronous work and far safer.

Meaningful readiness checks. Distinguish “process is alive” from “ready for traffic”. A readiness check that waits until critical connections and caches are prepared stops the load balancer from sending traffic too early.

Pre-scaling before reopening. If the platform scaled down during the outage, scale back to normal capacity before letting traffic return.

Reading the pattern during an incident

If a service recovers and immediately fails again in a loop, look at the incoming request rate at the moment of recovery compared with its normal level. A spike several times normal, followed by rising latency and then failures, points to a retry storm rather than a fresh bug. The quickest relief is usually to reduce incoming traffic: rate-limit at the gateway, temporarily disable non-essential callers or features, or pause queue consumers, then bring traffic back in stages. Restarting the service again, on its own, only repeats the cycle.

The broader point is that resilience is a property of the whole system, not of individual services. A service that handles 3,000 requests per second in steady state may handle far fewer in the minute after a restart, and the system has to be designed for that minute.

Written by the RP Strativa Team. Traffic figures in the examples are illustrative.

Leave a Reply

Trending