Start with the part that is easiest to lose while reading the rest: no money was taken. No wallet was debited twice, no card was charged, no subscription allowance was spent on work that did not happen.
What broke was a number on a screen. For some accounts the usage breakdown on your own account page read high — on the worst one we measured, 2.4 times the ledger it is supposed to summarise. The ledger was right the whole time. So were the monthly ceilings, which are read from somewhere else entirely. It was a display defect, it is fixed, and here is what caused it.
A key that looked for something nobody wrote
Work that happens because of something you did — updating a daily usage summary, crediting a referral, fanning out a notification — does not happen inside your request. The request writes a message and a background consumer picks it up a moment later, which is what keeps a slow side effect from making a fast page slow.
The hazard in that arrangement is doing the same work twice, because a broker will cheerfully deliver the same message again after a hiccup. The defence is an inbox: before processing, a consumer claims the message by writing a row carrying that message's key, and a unique index makes a second claim impossible.
Ours claimed on a pair — the tenant the message belonged to, plus the message key — and looked that pair up against the tenant id carried in the message envelope, which is populated.
The claim rows carried no tenant id at all. Every one of them was null and had been from the beginning. The entity's own documentation stated, in as many words, that the framework stamped it on insert. It never did. Measured against production on the day we found it: all 3,344 inbox rows null, while 1,174 of 1,451 rows on the producing side carried a real tenant — which is exactly why nobody suspected the reading side.
So the lookup asked for (tenant, key), the stored row was (nothing, key), and it could never match.
Nothing failed, which is why it ran for weeks
A lookup that can never match sounds like it should raise an error on the first try. It did the opposite.
The claim decided the message was fresh. The insert then hit the unique index and was rejected as a duplicate — correctly, because the row was already there. A fallback caught that rejection and read it charitably: another delivery must have claimed this one, nothing to do here. The envelope was left in its "still to process" state, so the next delivery picked it up and did the work again.
Every consumer was re-running its own backlog, roughly every forty-five seconds, indefinitely. Twenty-six of them were doing it. One production row finished 333 hours after it arrived, and had been finishing over and over for most of that time.
What it actually broke
Of those twenty-six consumers, four accumulate rather than overwrite. Those four are the billing projections the usage page reads, and they are where this surfaced.
They had an explicit protection against exactly this, written after an incident in August in which a single event was applied nine times to a gauge. It marks a row as succeeded before processing it, so a redelivery finds the mark and stops. The mark was written correctly every time. The next claim then looked for it with the same broken key and could not find it. The guarantee had been void since the day it shipped.
What this did not touch is worth listing precisely, because "billing projection" sounds alarming and the facts are not:
- The ledger was correct. Every AI call writes exactly one billing record carrying its own price snapshot, and nothing on this path writes those.
- No wallet moved. Balances come off the ledger, never off a summary.
- No ceiling moved. The budget guard reads a separate counter, which is the sole source of truth for what you have spent this month. Nobody was cut off early because of an inflated number, and nobody was let through who should not have been.
The only thing that reads the affected summary is the usage history on your own account page — your total, and the breakdown of what it went on. That is the whole blast radius, and what it did was read high.
The half nobody was looking at
The same broken lookup sat in a second place: the path that runs when a message fails. There it never found the row it was meant to update, and directly beneath it sat a branch carrying the comment "unreachable in practice" — which returned "this failure is terminal" on every failure.
So the retry count never advanced and no poisoned message ever reached the dead-letter queue. An entire layer of failure handling had been switched off behind a comment telling everyone who read it there was nothing here to check.
The fix
Both lookups now key on the message key within the consumer group, which is what the unique index enforces anyway. Message keys are derived from the id of the thing that produced them, so they cannot collide across tenants by construction.
The documentation that asserted the stamping was corrected rather than deleted. That sentence was the trap, and deleting it would leave nothing behind saying somebody finally checked.
Nine tests pin the behaviour. Six of them fail against the old code, which we confirmed by putting the old code back.
Cleaning up afterwards
The inflated rows repair themselves. A worker rebuilds the daily summaries from the ledger every hour, across a two-day window. It had been running the entire time and losing: once an hour against a consumer re-applying every forty-five seconds is not a fair race.
With the consumer no longer re-running, it wins. The two days still inside that window were rebuilt from the ledger once the fix was live. Two earlier days had already fallen outside it, and what remains on those is an account of our own plus one negligible difference elsewhere — so we left them rather than widen a production window to chase it, which seemed better than implying the sweep was total.
What we took from it
A deduplication key has to mirror what the index actually enforces, not what the code believes was written. The two halves of this one were fed from different fields and neither had ever been measured against the other. Both have been now, and the test that proves it goes red against the version that was wrong.