Picture an imaginary food-ordering app, call it QuickThali, on a Friday at 8 PM. Its checkout service calls a third-party payment gateway. At 8:02 the gateway starts responding very slowly, taking 30 seconds instead of one. QuickThali has set a 30-second timeout, a number someone chose years ago and never revisited.

Watch what happens next. Each checkout request now holds a worker thread for 30 seconds waiting on the gateway. The service has 200 threads. Orders arrive at, say, 50 per second. In four seconds all 200 threads are stuck. Now even requests that never touch payments, like viewing the menu, queue up with nowhere to run. The gateway’s trouble has become QuickThali’s outage. Nothing in QuickThali’s own code is broken.

A circuit breaker is the fix for this particular story. The name comes from the electrical device: when current is dangerously high, the breaker trips and cuts the circuit so the wiring does not burn.

Three states

The breaker is a small wrapper around the risky call. It remembers recent outcomes and sits in one of three states.

Closed is normal. Calls go through. The breaker quietly counts failures, where a failure means an error response or a timeout.

Open is the tripped state. Once failures cross a threshold, say 50 percent of the last 20 calls, calls are no longer attempted at all. They fail immediately, in a millisecond, without touching the gateway.

Half-open is the cautious test. After a cooling-off period, say 30 seconds, the breaker lets a few trial calls through. If they succeed, it closes again. If they fail, it opens again and waits another round.

Replaying the outage with a breaker

Back to 8:02. The gateway slows down, and with a tighter timeout of, say, three seconds the first calls start failing. Within a few seconds more than half of the last 20 calls have failed, and the breaker opens. From that moment, checkout requests that need the gateway get an instant failure instead of a 30-second hang. Threads are freed in milliseconds. The menu still loads. Customers see a message quickly, rather than a spinner.

Thirty seconds later the breaker goes half-open and sends three trial payments. The gateway is still sick; they fail; the breaker reopens. At 8:09 the gateway recovers, the trial calls succeed, and the breaker closes. Normal traffic resumes without anyone paging an engineer to restart anything.

There is a second benefit that is easy to miss. A struggling service needs less load to recover. A breaker that stops the flood of retries gives the gateway room to breathe, rather than hammering it the whole time.

What to do when the circuit is open

Failing fast is only half a design. The other half is deciding what the user sees. Options depend on the feature:

  • Return cached or slightly stale data, for example last known delivery estimates.
  • Offer a fallback path, such as a second payment provider or cash on delivery.
  • Accept the request and queue it for later processing, when the user need not know the result immediately.
  • Show an honest, specific error. “Payments are having trouble, please try again in a few minutes” beats a generic failure.

Some actions cannot be faked. You must not pretend a payment succeeded. Fallbacks that change money or legal records deserve far more care than fallbacks that change what a menu looks like.

Settings that need thought

A breaker has knobs, and bad values cause bad behaviour.

  • If the failure threshold is too sensitive, two unlucky errors on a quiet service trip the breaker for no reason. Libraries therefore usually require a minimum number of calls before judging.
  • If the open period is too short, you keep poking a service that has not recovered. Too long, and you stay down after it has recovered.
  • Not every error should count. A 400-style response because the customer typed a bad card number says nothing about gateway health. Count timeouts, connection failures and server-side errors.

Also keep one breaker per dependency, not one global breaker. If the recommendation service is failing, you do not want payments blocked.

How it fits with its neighbours

A breaker does not replace timeouts; it depends on them, because a call that never returns never registers as a failure. It also sits alongside retries and bulkheads, which are separate pools of threads or connections per dependency so that one slow call cannot eat everything. Retries and breakers need to agree: a retry loop wrapped inside a breaker multiplies the failure count, and a retry outside it should respect the open state. Libraries such as Resilience4j in Java and Polly in .NET package these patterns, and service meshes can apply them at the network layer.

Finally, expose the breaker state in your metrics and alerts. An open circuit is a loud signal that a dependency is unhealthy. If it trips at 3 AM and nobody ever finds out, you have built a mechanism that hides problems instead of surfacing them.

Figures and names in the examples are illustrative.

Leave a Reply

Trending