Our Rate Limit Said 100 a Minute. Behind 50 Gateways, It Was 5,000.

The token bucket took an afternoon. Getting fifty machines to agree on one count took three rebuilds.

(Fifty gateways, fifty honest counters, one cap nobody approved.)

The page that didn't say what I expected

My pager went off in the middle of a traffic burst, and not for the reason I'd have guessed. The alert was database CPU. One account had been flooding through a limiter I had personally promised could not leak.

The config said 100 requests a minute. We ran 50 gateways behind a load balancer. If that balancer spread one user's traffic across all of them, the real ceiling was 5,000 a minute, and no single box ever saw enough traffic to refuse.

I'd spent the first days of that project arguing with myself over which algorithm to use. That was never the problem. The problem was where the count lived once fifty machines had to agree on it, and I didn't see it until the pager did.

A rate limiter is really guarding the bill

Money is why this wakes people up. A rate limiter sits between one abusive client and a database bill you didn't budget for. It also sits between your free tier and the capacity your paying customers are actually renting. Set it too loose and a scraper runs up your spend or starves everyone else mid-incident. Set it too tight and on-call gets paged because real traffic is eating 429s.

We skipped the boring requirements, which is exactly where the leak started.

Pick an identity to key on. User ID first. API key if you must. IP only as a genuine last resort. Behind NAT, thousands of legitimate users can share a single IP address, so unless you normalise client IPs at your edge you'll block an entire office building because of one bad actor.
Put the check at the gateway, before requests fan out to your services.
Answer a rejection with a 429 and a Retry-After header, so clients back off instead of hammering.

An in-memory counter works until the second gateway boots

You start with an in-memory counter per user that resets each window. It's clean, it works, and then a second gateway boots. Now there are two counters that will never once agree.

The token bucket is the elegant single-machine version. Refill tokens at a fixed rate, spend one per request:

single-machine only; a distributed version needs shared state

elapsed = now - last_refill

tokens = min(capacity, tokens + elapsed * refill_per_sec)

allowed = tokens >= 1

if allowed:

tokens -= 1 # spend one, carry the rest

last_refill = now # persist this AND tokens, or elapsed grows forever

The other algorithm worth knowing is the sliding-window counter, which blends the current and previous window so a user can't fire a full quota on each side of a window boundary.

Why not the others:

Fixed window is the simplest thing that counts, and it bursts at the edges. A user spends their full quota in the last second of one window and the full quota again in the first second of the next, so a 100-a-minute cap waves through 200 requests in about two seconds.
Leaky bucket smooths output to a constant rate by queueing requests. That's a queue with extra steps, and on an HTTP path you almost never want to hold a request open when you could reject it cheaply.
Sliding-window log stores a timestamp for every request and counts the ones still inside the window. It is exact, and its memory cost grows with your traffic. That's precision you rarely need at a price you'll definitely feel.

The problem lies entirely in tokens and last_refill. They are state, sitting in local memory. Put that behind 50 gateways and you get 50 buckets, each cheerfully waving through 100 requests as though it were the only one in the room. The maths is correct. The state is in the wrong place, and that's a distributed systems problem wearing an algorithm's clothes.

The 99 + 99 race: two gateways, one Redis counter, both say yes

So you move the count into Redis. One source of truth, every gateway reading and writing the same key. That's the right instinct, and it still breaks.

The code below uses a plain integer counter rather than a full token bucket. The race is easier to see in a single variable, and a real bucket has the same hole with more arithmetic on top of it.

(Two gateways read the same 99, both decide "under the limit," and the shared counter ends up past the cap.)

A user sits at 99 requests this window. Two more requests hit two gateways at the same moment, each reading the counter before the other one increments it. Gateway A reads 99. Gateway B reads 99. Both see a number under 100. Both say yes. Both increment. The counter lands on 101, and a request got through where the cap should have stopped it.

A limiter that isn't atomic isn't a limiter. It's a strongly-worded suggestion.

Multiply that across every user pressed against their limit during a burst and your hard cap goes soft exactly where you were counting on it. It's the same bug as count++ from two threads, except the race now runs across machines instead of cores, which makes it easier to miss and worse when it lands.

The GET-then-INCR bug I shipped, and the Lua that fixed it

Here's what I actually put into production:

key = f"rl:{user_id}" # one counter per user

count = redis.get(key) # gateway A and B both read 99

if int(count or 0) < 100:

redis.incr(key) # both pass the check, both increment

allow() # both allowed, one request over the cap

I swore this was safe because Redis is single-threaded. That part is true. Redis runs each command atomically. It just doesn't run my three round trips as one command, and the gap between the read and the write is where the whole thing lives. Under load, something always slips into that gap. The customer sailing past their cap wasn't a customer bug. It was my mental model, rendered in production traffic.

The fix is to make check-and-increment a single thing Redis cannot interrupt: a Lua script, run with EVAL. Read the counter, compare it to the limit, increment, set the window, all server-side, one round trip.

-- EVAL runs this block atomically; nothing interleaves, not even expiry

local n = tonumber(redis.call('GET', KEYS[1]) or '0')

if n < tonumber(ARGV[1]) then

if n == 0 then

redis.call('SET', KEYS[1], 1, 'EX', ARGV[2]) -- new window: set the TTL once

else

redis.call('INCR', KEYS[1]) -- existing window: just count

end

return 1 -- allowed

end

return 0 -- 429; the race is gone

I got the expiry wrong on the first attempt, so learn it from me instead. Set the TTL only when the window is created, on n == 0. Refresh it on every request and the window never rolls: "100 a minute" quietly becomes "100 until the user goes quiet for a full minute," which is a maddening thing to debug at 2am. If a key ever loses its TTL through a manual write or an old migration, it counts forever, so re-assert the expiry defensively.

That EX leans on Redis's own expiry rather than any gateway's wall clock, which is a relief across fifty boxes. Don't oversell it, though. Expiry is fine for limiter windows and it is not a precise timestamp source. Clock skew bites the other common design, where each gateway builds a per-minute key like rl:user: from its own clock and drift smears one user's requests across two windows. If you build it that way, call redis.call('TIME') inside the script and let one clock decide, rather than trusting fifty.

Four questions I'd ask before trusting any limiter

Yours, someone else's, or one being sketched on a whiteboard in an interview. These are what I actually check:

Is every check-and-increment a single atomic operation? If you can point to a moment between the read and the write, you have a race.
Whose clock decides the window? One shared source, or fifty boxes each guessing at "now"?
When Redis is unreachable, does this endpoint fail open or fail closed? By decision, not by whatever your client library happens to default to.
What is your actual ceiling? The number you published, multiplied by the number of things that can enforce it independently.

One global counter is correct, and too slow to live with

Moving to a single Redis fixed correctness, and then latency arrived to collect. Every request in Frankfurt paid roughly 90ms to ask a box in us-east-1 for permission. A fixed round trip like that doesn't just fatten the tail. It lifts the whole floor, p50 included. The answer was right and unshippable on a hot path.

Which brings you to quota leasing, where "global throttling" stops meaning one scorching-hot counter. You carve the global budget into leases and hand each gateway a slice to spend locally, at memory speed. A gateway phones home to top up only when it runs low, say under 20 percent, so it never blocks a live request on the network.

One catch is worth naming. If you hand every region an equal slice of the quota, your busy regions will reject legitimate traffic while your quiet regions sit on a budget they'll never touch. Size the leases by recent demand, or let a region claw back what its neighbours aren't using.

(Leasing splits a global budget into slices spent locally at memory speed, so gateways phone home to renew rather than on every request.)

The tradeoff is clean. Worst-case overage is lease size times leaseholders, so ten gateways on a 20-request lease can drift about 200 over before anyone notices, and a gateway that dies mid-lease strands its slice until it expires. You're buying back a lot of latency with a little exactness, knowingly.

The incident itself only cost us an apology and a same-day patch. It stung, we lived. The rebuild is what hurt. We built this three times before it held, counters to Lua to leasing, each pass because we hadn't thought hard enough the round before. Leasing also left more moving parts to watch than a single counter ever asked for.

Match the failure mode to the blast radius

Fail-open versus fail-closed is a per-endpoint call, not a global switch, and getting it backwards buys you a separate outage. In code you wrap the Redis call in a try/except with a timeout, so a slow Redis can't hang the request. The choice inside that except block is the entire decision.

Login and payments: fail closed. A slow Redis shouldn't hold the door open for an attacker. If a hard deny is too blunt, drop to a smaller local allowance or fall back to a cached allowlist.
Read-only content feeds: usually fail open. Blocking real people over a two-second Redis blip is worse than letting a trickle past.
The bigger the blast radius, the harder you scrutinise that default.

The single global counter is also a single point of failure, which is the same question one layer down. Run Redis with replicas and every failover forces a call on the in-flight counts: whether they survive, get double-counted on replay, or get thrown out to be safe. We ducked that call, and the failover made it for us: the replica promoted with empty counters, and we waved a large traffic spike straight through to the database. Once burned, we settled on deny-on-failover for the auth counters and a little tolerated overage on content.

Then wire the alerts, not just the graphs:

Allowed versus denied requests, per endpoint
Redis latency and timeout rate
Requests that slipped past the cap
Any lease still holding budget it isn't spending
A page when the fail-open count spikes

If you don't track these metrics, your limiter will fail open silently, and your first warning will be a crashed database.

Don't reach for leasing because it looks sharp in a design review

For a single-region product, the honest answer is one Redis, one Lua script, fail-closed on the auth paths, and stop. In-region, that round trip is sub-millisecond. Upgrade the counter to a sliding window if the boundary burst starts biting. Leasing only earns its keep when cross-region latency is genuinely hurting and someone has signed up to own the alarms.

Every stage bought something and cost something. The local counter leaked quota. The global counter plugged the leak and put a network hop in front of every request. Leasing bought the speed back and handed us a pager full of stranded-lease alarms. The job is knowing which downside you can stomach, and being able to say out loud why the counter lives where it does when someone asks.

The hard part was never 100 requests a minute. It was proving there is exactly one place allowed to count them, and keeping a graph that says it still is.

Thank you for reading. If you found this useful, follow PracHub for more practical deep dives on engineering, system design, and interview preparation written to help you build skills that last beyond a single article.