Suppose you have built a small API that sends order-status updates to a few partner companies. One partner writes a buggy loop and starts calling you 5,000 times a second. Your database slows down, and every other partner suffers. Rate limiting exists to stop exactly this, and the idea is easy to say: allow each caller only so many requests in a given time. The details are where it gets interesting, so let us build it up slowly.

Start with the naive counter

The first thing most of us think of is a fixed window. For each caller, keep a counter. At the start of every minute, reset it to zero. Each request adds one. If the counter passes 100, reject with HTTP status 429 (Too Many Requests).

It is cheap, it is easy to explain, and it has a flaw you should see before you ship it. Imagine the window resets at 12:01:00. A caller sends 100 requests at 12:00:59 and another 100 at 12:01:00. Both batches are legal, yet your server absorbed 200 requests in about two seconds, double the limit you thought you had. The limit is correct per window and wrong per any other interval you care about.

Sliding windows: fairer, a bit costlier

A sliding window log fixes this by remembering the timestamp of every recent request. When a new one arrives, throw away timestamps older than 60 seconds and count what is left. It is exact, but you store up to 100 timestamps per caller, which hurts when you have a million callers.

A cheaper compromise is the sliding window counter. Keep two counters: the current window and the previous one. Estimate the rolling count as the current count plus the previous count multiplied by how much of the previous window still overlaps. If you are 15 seconds into the current minute, 75 percent of the previous minute still overlaps. With 80 requests last minute and 30 so far this minute, the estimate is 30 + 0.75 × 80 = 90. It is an approximation, since it assumes last minute’s requests were evenly spread, but it is usually good enough and needs only two numbers per caller.

Token bucket: allowing bursts on purpose

Real clients are bursty. A mobile app opens a screen and fires six calls at once, then does nothing for ten seconds. A strict limit of “one request every 600 milliseconds” would punish that harmless behaviour.

The token bucket models it differently. Picture a bucket that holds at most 10 tokens. Tokens drip in at 2 per second until the bucket is full. Every request takes one token; if the bucket is empty, the request is rejected or delayed. So a caller who has been quiet can burst 10 requests instantly, but over a long period can sustain only 2 per second. Two numbers, capacity and refill rate, describe the whole policy: how big a burst you tolerate and what the long-run average is.

You do not need a timer that adds tokens. Store the token count and the time of the last update. On each request, work out how long it has been, add elapsed × rate tokens (capped at capacity), then subtract one. That is constant work and constant storage per caller, which is why token buckets are so widely used in gateways and cloud services.

A close relative is the leaky bucket, which queues requests and releases them at a fixed pace. It smooths traffic into a steady stream, which suits a downstream system that cannot tolerate spikes, but it adds waiting time instead of rejecting.

Who is the “caller”?

The algorithm is the easy half. Deciding what the limit is keyed on is harder.

  • An API key or logged-in user ID is the fairest key, but it only works after authentication. Unauthenticated endpoints such as login or signup still need protection.
  • An IP address works before login, but many people can share one. A whole office, a college campus or a mobile operator using carrier-grade NAT can appear as a single address, so a tight per-IP limit can lock out hundreds of innocent users.
  • Behind a load balancer or CDN, the address your server sees is the proxy’s. You must read the forwarded-for header, and only trust it if it was set by your own infrastructure, since a client can send a fake one.

Many systems layer limits: a generous per-IP ceiling to blunt crude floods, a per-user limit for fairness, and a tight limit on sensitive actions like OTP requests or password attempts.

When you have ten servers

If each server keeps its own counter, a caller allowed 100 requests per minute can really make up to 100 per server, depending on where the load balancer sends them. Two common fixes exist. One is a shared store such as Redis, where the check and the update must happen as a single atomic step, otherwise two servers can both read “99” and both let a request through. The other is to divide the limit locally (10 servers, 10 each) and accept some unfairness. The shared store is more accurate but adds a network hop to every request, and you must decide what happens if the store is down. Failing open keeps the service alive but unprotected; failing closed protects the backend but blocks everyone. Both are defensible, and the choice should be made on purpose, not discovered during an outage.

Being kind to the caller

A good limiter tells clients what happened. Return 429, include a Retry-After header with the number of seconds to wait, and document the limits. Many APIs also send headers showing the remaining quota. Well-behaved clients then back off, ideally with a little randomness, instead of hammering you in lockstep the instant the window opens.

One last point: rate limiting protects fairness and basic stability, but it is not a complete defence against a determined attacker, who can spread requests across thousands of addresses. Treat it as one layer, alongside authentication, quotas and capacity planning.

Figures and names in the examples are illustrative.

Leave a Reply

Trending