Here is a failure that confuses people the first time they see it. A payment provider starts answering in 10 seconds instead of its usual 300 milliseconds. It is not down, just slow. Within a few minutes the entire checkout service stops responding, including endpoints that never call the payment provider: the cart page, the address book, even the health check. Nothing crashed. The service simply ran out of something every request needs, and the slow dependency was holding all of it.

The arithmetic of waiting

Little’s law, a basic result from queueing theory, says the average number of items in a system equals the arrival rate multiplied by the average time each item spends there. Applied to a service, the number of requests in flight at any moment equals requests per second multiplied by seconds per request.

Take an illustrative checkout service running on Tomcat, whose default maximum is 200 request-processing threads. It receives 200 requests per second in total, and 20 of those per second call the payment provider. Normally each payment call takes 0.3 seconds, so on average 20 × 0.3 = 6 threads are busy waiting for payment. Hardly noticeable.

Now the provider slows to 10 seconds. The payment path alone needs 20 × 10 = 200 threads to keep up. That is every thread the server has. Cart requests, which need a thread for only 20 milliseconds, queue behind them and time out. The load balancer’s health check also needs a thread, fails, and may pull the instance out of rotation, pushing more load onto the remaining instances, which then fail the same way. A slow dependency has become a total outage through resource exhaustion.

The same thing happens with any shared, finite resource: database connection pools, outbound HTTP connection pools, memory used by queued requests, or file descriptors. Whatever is shared and bounded will be filled by whoever holds it longest.

Defence one: timeouts that mean something

The first line of defence is a timeout on every remote call. It sounds obvious, but many clients default to waiting indefinitely. Java’s older HttpURLConnection uses zero, meaning no timeout, unless you set one. Python’s popular requests library waits indefinitely unless you pass a timeout. A missing timeout is a latent outage waiting for a slow dependency.

Set both kinds: a connect timeout (how long to wait to establish a connection, usually short, around a second or less) and a read or response timeout (how long to wait for data). Choose the read timeout from the dependency’s real latency distribution, typically a bit above its p99, and from your own deadline. If your checkout endpoint promises a response within 3 seconds, a 10-second payment timeout makes no sense.

Timeouts reduce the time a thread is held, which by Little’s law directly reduces how many threads the slow path can consume. With a 2-second timeout, the payment path in the example holds at most about 40 threads, leaving the rest for everything else. But 40 threads are still being spent on calls that will fail, and if the arrival rate rises the problem returns. Timeouts limit the damage; they do not isolate it.

Defence two: bulkheads

Ships are divided into watertight compartments called bulkheads, so a breach floods one section rather than sinking the vessel. The software pattern, popularised by Michael Nygard’s book Release It!, applies the same idea: give each dependency its own limited share of resources, so exhausting one share leaves others untouched.

Common ways to build a bulkhead:

  • A separate thread pool per dependency. Payment calls run on a pool of, say, 30 threads. When all 30 are busy, further payment calls are rejected immediately instead of borrowing request threads.
  • A semaphore per dependency. Lighter than a pool: a counter that allows at most N concurrent calls to the payment provider and rejects the rest at once.
  • Separate connection pools per downstream service, with their own maximums, rather than one shared HTTP client for everything.
  • Separate deployments. At a larger scale, run payment-related endpoints on different instances from browsing endpoints, so even a complete exhaustion of one set cannot touch the other.

Resilience4j provides both thread-pool and semaphore bulkheads for Java. Whatever the tool, the key decision is what to do when the compartment is full. Usually you fail fast with a clear message, such as “payment is taking longer than usual, please retry”, rather than queuing indefinitely.

Defence three: stop calling a dependency that is not answering

Even with timeouts and bulkheads, each call to a sick provider still waits for the full timeout before failing. A circuit breaker watches the failure and timeout rate, and once it crosses a threshold, fails calls instantly for a cooling period, then tests the dependency with a few trial calls before resuming. This turns slow failures into fast ones, which frees resources and gives the provider breathing room.

Does async I/O make this go away?

Partly. Event-loop servers like Node.js, or Java services using reactive frameworks or virtual threads, do not tie up an operating system thread per waiting call, so the 200-thread ceiling disappears. The underlying problem does not. Waiting requests still hold memory, sockets and often database connections, and they still pile up according to Little’s law. An async service facing a 10-second dependency simply accumulates more in-flight requests before something else runs out. Timeouts and concurrency limits remain necessary.

Designing the business flow for slowness

The deepest fix is often in the product flow, not the infrastructure. Does the user have to wait synchronously for the payment provider at all? Many checkouts create the order in a “payment pending” state, hand off to the provider, and confirm through a callback or webhook, with a status page that polls. Indian UPI collect flows already work this way, because the customer approves the payment in a separate app. When the provider is slow, the order simply stays pending a little longer, and nothing else in the service is affected.

The checklist that comes out of this is short. Every remote call has a timeout shorter than the caller’s own deadline. Every critical dependency has a bounded share of resources. Calls to a failing dependency fail fast. And the user-facing flow, where possible, does not depend on a third party answering quickly.

Written by the RP Strativa Team. Traffic figures and thread counts in the examples are illustrative.

Leave a Reply

Trending