A Check That Could Never Say Yes

Every event our platform processes — a payment recorded, a referral bonus earned — goes through a queue built on one promise: handle each message once, no matter how many times it gets delivered. The check that keeps that promise had been silently useless for weeks, and nothing about it looked broken.

A comparison that could never agree

The check works by comparing an incoming message against a saved record of messages already handled, matching on which tenant the message belongs to plus its own key. The incoming message always carries a real tenant, read off a header the sending side writes. The saved record's tenant was supposed to be filled in automatically the moment the row was written — the code's own documentation said exactly that — and it never was. Every affected row, checked today, carries no tenant at all.

So the comparison was between something and nothing. For any event tied to a tenant, it could never match.

Why it looked like nothing was wrong

A message that collides with a record already claimed is supposed to stop there. Here, every attempt inserted a fresh claim, every insert collided with the one already sitting in the database, and the code path that handles that collision assumed the obvious explanation — another delivery got there first — and quietly sent the message back to be processed again. One message we found had been picked up roughly every forty-five seconds, continuously, for close to two weeks before it was finally marked done.

The failure-handling side was worse. The same broken comparison decided whether a failed message should be retried or given up on. Since it could never find the row it needed, every failure was treated as unrecoverable — except the line of code that should have quarantined it had been marked, in a comment, as unreachable in practice. It wasn't unreachable. It ran on every single failure, and nothing was ever set aside for a closer look.

What it touched, and what it didn't

The queues that carry billing events feed the running total your spend history shows for a given day. That's the surface this hit: a message reprocessed nine times updates that total nine times. On the worst day we found, the total shown counted forty-seven completed charges where the underlying ledger held thirty-seven — about 2.4 times over.

The charge itself was never repeated. What actually decides whether you can spend — your budget, your balance — reads from a separate source this bug never touched. This was a display problem: a number on a page disagreeing with the ledger behind it, never money moving twice.

The fix, and what we chose to leave

The comparison now keys on a message's own identity inside the queue it belongs to — which is exactly what the database was already enforcing, once every stored row turned out to carry no tenant. That identity comes from the record the message describes, so it cannot collide across tenants by construction. Nine new tests pin this down; six of them fail if the old comparison is put back.

A separate job already recomputes each day's total from the ledger on a rolling two-day window, so the fix alone caught most of the damage without any manual repair: one of the two most recent days came back exactly right, the other landed with the money already correct and only a single stray transaction left in the count — close enough that the next scheduled pass clears it. Two earlier days sit just outside that rolling window and still show the same small kind of gap. Fixing them means widening the window or running the job by hand for a slice of history nobody is actively looking at, so they were left as a known, bounded exception rather than forced shut.