One Consistency Level Broke Our Store. Now Every Route Gets Its Own Choice.

I handed the same consistency rule to a product page and a checkout. One needed to stay available, the other needed to be exactly right, and I'd given them both the same thing.

(One consistency label, stamped onto every route. The product page got the same treatment as the checkout, and that was the mistake.)

I painted CP across the whole store, and the product pages went dark

Two regions of our store, one customer in each, both mid-checkout for the last unit of the same SKU. I assumed the hard part would be deciding who got it. Instead the whole storefront went dark for everyone else, because a product page couldn't fetch a name. I'd painted "CP, strong consistency" across every route without once asking what that meant for any single one of them.

A read that could happily have served a three-second-old name returned nothing instead. Customers got 503s and left. Every one of those 503s was my own instruction, carried out faithfully, and that is the part I keep coming back to. The disaster everyone braces for, two regions each selling that same last unit, never happened. The near-miss did. The fix that would have caused it was already sitting in a branch, one approval away from production. If a reviewer hadn't asked the right question, I'd have shipped it.

Pick two is the wrong question in production

The three circles never told me the thing that mattered, and I stared straight at them while the 503s climbed. Partition tolerance was never mine to grant or refuse. A switch rebooted upstream, the two regions stopped talking, and my whole job became triage: what breaks, and where. The diagram made that look like a single checkbox, one word, "CP", done. But the real decision isn't permanent and it isn't global. You get a fresh one per operation, only while the split lasts, and then the link heals and you forget it was ever a question.

(You never opt out of partition tolerance in a live system, so the only real choice is consistency or availability, per operation, while the split lasts.)

The design doc where I wrote one word and broke two things

In the design review someone asked if we were going CP. I said yes. That was the whole conversation. One word for the entire service, and nobody, me least of all, stopped to ask what it did to a product page versus a checkout.

When the link dropped, every read that needed cross-region quorum failed, product pages first. The culprit was the plain product lookup. It was running at the cluster-wide strong default, which meant it had to reach the remote datacentre before it could return a single name:

// product read, inheriting the cluster-wide strong default:

session.execute(readProduct, ConsistencyLevel.QUORUM);

// global QUORUM must reach the remote DC; during the split it can't, so every product page 503s.

A message landed in the channel, like we'd flipped one wrong switch: "So is it CP or AP?" Neither, I said. It's whatever you set it to, per query.

So I set it. A name can sit a few seconds out of date without anyone noticing, so I scoped product reads to a quorum inside the local datacentre instead of across both. The 503s drained on the next deploy:

// a name can be a few seconds stale, so don't cross an ocean to render it:

session.execute(readProduct, ConsistencyLevel.LOCAL_QUORUM);

// keeps quorum inside the local DC, so reads stay up in-region while the two regions can't talk.

That change solved the immediate problem, but it hid another one. Scoping quorum to the local datacentre was an availability move, plain and simple. What keeps that local read consistent is a separate rule, R plus W greater than N, which just means the set of nodes you write to and the set you read from are guaranteed to overlap, so a read always catches the latest write. I confused the two for months. And even with the dial set right, read repair and clock drift can still bite, so tuning Cassandra is something you keep watching, not a switch you throw once.

Then I almost swung the pendulum too far the other way. My plan was to flip everything to available, which meant letting both regions accept inventory writes locally. A reviewer asked the plain question: what happens to the stock count when both regions decrement it with nothing coordinating between them? There it was, the oversell I'd dodged in production, sitting quietly in my own branch. decrementStock runs twice, the single unit sells twice, and the warehouse finds out days later from a spreadsheet that doesn't add up. Checkout went the other way. Instead of promising inventory it couldn't prove it had, it answers with a 409 Conflict, the HTTP status for a clash with current state, and lets the client retry:

// the last-unit path stays strict during the split:

if (!reserveWithQuorum(sku, qty)) return status(409);

It took me the better part of a year to see that my second mistake was a language bug. Sitting there looking at that inventory path, I'd told myself "strong consistency" was what kept the counter from going negative. It wasn't. All it ever gave me was the freshest possible reading of a number that could still say minus one, and a perfectly current -1 is still -1. What actually stops the sale is an invariant, a constraint or a conditional reservation write, and I'd never written one.

Four things a customer can see when the link drops

In the postmortem I read the traces instead of arguing about the database. Not one of the four behaviours had been picked on purpose. Product pages rejected reads when they should have served a stale name. The inventory path had come within a single review comment of the one move it could never survive. And the behaviour the catalogue actually needed, serve the old name and stay up, was written down nowhere. We'd backed into every last bit of it.

When the network splits, your system falls back to one of four behaviours, whether you picked it or not. It can wait, holding until it can guarantee consistency, and the user watches a spinner that either resolves or times out. Rejecting is the cleaner failure: a 503 with a Retry-After tells the client exactly what happened and what to do next. Serving stale keeps the lights on with a slightly old read. And accepting the write on both sides defers the argument to later, a 202 that means "we took this", not "this is settled", with the merge left to last-write-wins or to a human with a spreadsheet.

We were treating a complex product like a single database toggle. The product page needed to serve stale and I'd forced it to reject; inventory needed to reject and I'd very nearly forced it to serve.

The stale question cuts finer than I'd realised. Nobody blinks at a stale name. A stale price is a support email, and a stale stock count is a refund, because a checkout a few seconds behind sells things that don't exist. Displayed prices can lag if you revalidate them before you charge; a stock count that authorises a sale can't. I'd treated all three as the same problem.

(One cluster can end up using all four responses across different endpoints, on purpose or by accident.)

CAP explained the outage. PACELC explained why the calm days were already slow.

The slow product pages I'd been grumbling about for months? That was the same mechanism as the outage. Every strong cross-region read was paying for a round trip to the far datacentre, just to fetch a name that changes maybe twice a year. That's the half CAP never mentions. PACELC does. Its "else" clause is the ordinary day, no partition anywhere, and you're still trading latency against consistency on every single read.

The four questions I now walk every route through

The first time I ran this on someone else's route, I asked what it returns when the link drops. "A spinner?" the dev guessed. It was a 503. Nobody had designed that behaviour; it had simply shipped, and the person who built it had no idea it was misbehaving. We changed its default, and the next time a link dropped the user got a Retry-After instead of a spinner that spun forever. Now nothing of mine reaches a multi-region cluster until it survives the four questions I never thought to ask the first time round.

What does the customer actually see when the link is cut: a spinner, an error, a stale read, or a 202 we might not end up honouring? How much does being wrong cost here, given a stale name is a shrug and a double-charged card is a refund? Is this a read or a write, keeping in mind that most catalogue reads can go stale but anything guarding money or access cannot? And when the split heals, who does the reconciling, a cron job or a person squinting at two conflicting rows late on a Friday night?

The answers don't live in my head anymore, which was the whole problem the first time. Every route now ships with its behaviour written down where a reviewer can't miss it:

GET  /product   -> serve stale, max age 10s                            owner: catalog

POST /checkout  -> reject 409 + retry contract                         owner: payments

POST /cart      -> accept 202, merge: union items, last-write-wins qty  owner: orders

The retry contract there is just the documented promise of when a client may retry and how often, so nobody has to guess.

What it cost, and why it isn't finished

We paid for this in meeting time. Every new endpoint gets a walkthrough now, and every walkthrough is an argument with someone who wants to just leave the default. One developer called it bureaucracy. I called it the price of not going dark again. Product reads moved in-region, so they can lag a few seconds. One customer emailed about a stale price; I told her it had refreshed two seconds after she saw it, and that the alternative was no store at all. She said she'd take the two seconds. The inventory path got slower and stricter too, so during a split you now get a real 409, and someone had to write and test the retry that handles it. I wrote "CP" in a design doc and then spent months unpicking what it meant for the product page, the checkout, and the warehouse.

A few months later, another link dropped. The losing region returned 409s, the client retried, the warehouse stayed honest, and checkout held. I don't get to feel much cleaner than that in this job, so I took the win and moved on.

Don't do this on a single Postgres. Your system can't have a distributed failure yet, so you'd just be drawing diagrams for the fun of it. But with real multi-region traffic and a named owner ready to reconcile when the split heals, this is where it pays off.

The route table is a hard gate now, not a polite suggestion. A route that hasn't declared what it does during a partition doesn't merge, and the check runs in the pipeline, not whenever someone remembers to look. One route slipped through before the gate existed, and I still can't leave it alone. Another team had shipped a /fulfilment route on the old strong default, back before the gate was there. I left a comment asking what a customer sees when the two regions can't talk to each other. It's still there. Three months. Nobody has answered the one thing I actually needed to know.

---

Thank you for reading. If you found this useful, followPracHubfor more practical deep dives on engineering, system design, and interview preparation written to help you build skills that last beyond a single article.