Load Balancing vs Consistent Hashing: The 80% Cache Mistake

Load balancers spread traffic; consistent hashing owns state. I confused the two, and one new node remapped 80% of the cache, burying the database it existed to protect.

[Add a fifth slot and almost every letter has to move. That's % N when N changes.]

One new node remapped 80% of my cache

We added a node to the cache pool for headroom. Read latency got worse. The hit rate cratered. Reads that used to land in memory hit the database all at once, and its CPU pegged. For a while we were sure it was the deploy. It wasn't the deploy.

Nothing was broken. The code did exactly what I told it to. I'd sharded the cache with hash(key) % N, and when N went from 4 to 5, the modulo result changed for most of the keys. In this 4-to-5 resize, about one in five keys kept the same node. The other four in five became misses at once.

The real mistake, though, was treating two different jobs as a single problem.

The diff passed review because nothing in it looked wrong. That's the part that stuck with me. I couldn't point at a bad line anywhere. The thing that broke us was an assumption I'd never written down: that N would hold.

I ran load balancing and consistent hashing as one job

The first job is spreading live traffic across interchangeable workers. Any healthy box can serve the next request, so you just want them evenly busy. That's load balancing, and it's stateless: nothing about it decides which node durably owns a key.

The second job is the opposite instinct: deciding who holds a piece of state when the cluster changes size. Which node owns this cache key, this shard, this session. You want the same key on the same node even after membership shifts. That's consistent hashing.

And I'd grabbed the first tool for the second job. % N spreads keys evenly across a fixed set of nodes, right up until the set changes. And adding a node is exactly that.

Confuse the two and every read request for a remapped key turns into a fresh database read. A miss storm aims your entire read volume directly at the database. You're load-testing your database with live traffic.

Choosing % N in the first place came with a comment I'm not proud of, something close to "we'd basically never change N." That promise lasted until month two.

Why load balancing wasn't the hard part

The traffic side was never what broke us; it's the half most teams handle fine. Which box answers a request is a traffic question, and I had a state question: who owns this key. No routing rule answers that. The moment a key had to stay put, load balancing was the wrong tool, and % N was the mistake already in production.

How consistent hashing moves a slice, not the whole cache

Consistent hashing fixes this by putting nodes and keys on the same circle. Hash every node ID to a point on it, and every key to a point too; a key belongs to the first node you reach going clockwise. Kill node 3, and only the keys on its arc move to the next node clockwise. The rest of the ring doesn't notice. A membership change now touches a fraction of your keys, not most of them, and virtual nodes keep those slices even enough to trust.

[Node 3 fails and only its arc of keys hands off clockwise to node 4. The rest of the ring never notices.]

The ring worked. One node still burned.

We shipped the ring. One key pinned a single node at 100% while the rest sat idle. Priya, our SRE, pulled the per-node graph, pointed at one key drawing orders of magnitude more reads than everything else, and made the point I still think about: the ring balances keys, not traffic, and this key only lived in one place.

Same mistake both times. First I trusted the node count to hold. Then I trusted balanced keys to mean balanced traffic. The ring spreads keys evenly; it says nothing about one key soaking up the reads of a thousand others.

Fixing it with a hash ring and hot-key salting

The original sharding line looked harmless:

node = nodes[hash(key) % len(nodes)]  # add a node, len(nodes) 4 -> 5, and ~80% of keys remap

(Separate gotcha: never use Python's built-in hash() for persistent sharding. For strings it's randomised per process, so it isn't stable across restarts. Reach for murmurhash or xxhash.)

A raw ring is lumpy: with one point per node, a few boxes can hash close together and hand a big chunk of the ring to one of them. Virtual nodes fix that by giving each node many points:

ring = ConsistentHashRing(nodes, vnodes=128)  # more points per node = smoother arcs; the count is a knob

node = ring.get_node(key)

The hot key the ring couldn't help needs a different fix: salt it so one key lives as several copies. Salting fans reads out but multiplies your writes by the replica count, and that's where you pay:

write: every update now hits all REPLICAS copies

for i in range(REPLICAS):

copy_key = f"{hot_key}#{i}"

ring.get_node(copy_key).set(copy_key, value)

read: pick any one copy

copy_key = f"{hot_key}#{random.randint(0, REPLICAS - 1)}"

value = ring.get_node(copy_key).get(copy_key)

[Salting a hot key splits it into copies across nodes. Every write hits all of them; a read picks one.]

So salting hands you a choice with no clean exit: pay the write amplification, serve stale reads, or rebuild it asynchronously in the background and hope the lag doesn't break an invariant. Skip it and you've traded a hot node for silent inconsistency, which is worse.

What the incident cost

Stopping the storm cost a month of running the old and new schemes in parallel, writing to both and reading from both, while the cache warmed. On top of that came a rewrite across every client that touched the pool. Worst of all was admitting that N had never been static. The salt layer left a standing chore too. A key lands on the hot list once its own reads per second cross a threshold, and that line moves as usage shifts, so someone always owns chasing it.

How Cassandra and HAProxy handle it

None of this was novel. The industry had already turned these lessons into knobs. Cassandra tuned its vnode count in the open, dropping the num_tokens default from 256 to 16 in 4.0 as its allocator got smarter. HAProxy has a setting called hash-balance-factor that caps how hot any node can get and spills the overflow onward, though that solves traffic skew, not the problem of replicating stateful keys. The math even crosses over. Google's Maglev runs consistent hashing inside its load balancers, but it places connections, not cache keys. Same trick, different object.

When to use consistent hashing, and when % N is fine

A ring earns its complexity when your node set churns at runtime and keys have to stay put, like sharding a session store or a cache that auto-scales. If that's you, use the ring. If not, you're buying machinery you'll maintain more than you use.

And the conventional wisdom, "use % N if your node set is static," deserves some suspicion. Modulo maths is fine for a one-off batch script or a small internal cache. But for a live production service, a "fixed" cluster size is a lie waiting to page you. Fixed clusters still lose and replace nodes, and they scale the week a deadline lands. Every one of those is a membership change in disguise. A ring doesn't make a design safer on its own. It helps only when churn is the failure you're solving, so weigh it against how much a membership change can actually break.

The ring shrank the problem into things we could watch. Hot-key rot, vnode skew, per-node load that finally meant something. You stop firefighting remap storms and start watching what actually moves.

A flat per-node request graph doesn't mean your load is balanced. It just means your traffic is. Look for the node quietly burning up.

Four questions before you reach for a hash ring

To avoid finding that out the hard way, run these before any ring code ships. Does the request need a specific machine, or will any healthy one do? If any will do, you want a load balancer, and a ring is complexity you'll regret. When a node joins or leaves, how many keys may move? A slice, or most of them? "Most of them" rules out % N by itself.

The third is the one I skipped. Is your load even because your keys are even, or are you assuming those are the same thing? When they drift apart, your per-node key chart shows a tidy balance while one box quietly burns. And settle the last one early: what happens the day one key soaks up more traffic than a thousand others? Decide now, while it's still a design question and not an incident.

Every scaling fix trades one bottleneck for another. I stopped chasing a cluster that couldn't break, and started building one where I'd see the break coming before anyone paged me.

If you want to learn from production failures before they hit your own cluster, follow The Speed Engineer in Beyond Localhost for more on system design, high-performance backends, and the engineering decisions nobody warns you about.

Thank you for reading. If you found this useful, followPracHubfor more practical deep dives on engineering, system design, and interview preparation written to help you build skills that last beyond a single article.