Three Pages a Second

On the morning of 22 September, torun.ai slowed to a crawl. A farm of automated browsers, more than two thousand addresses in three quarters of an hour, was requesting our public pages one at a time, and at the worst minute it asked for about three pages a second. A website should shrug that off. Ours did not: in the busiest window more than a quarter of requests took over thirty seconds, and hundreds of visitors gave up before a page arrived.

Not the load we assumed

The obvious suspects were the crawler and the processor. Neither held up. When we replayed the same three pages a second against a copy of the site, it kept up for less than a minute and then stopped completing anything at all. Processor use fell from 60 percent to 1 while the number of threads climbed from 12 to 166. A server that is overloaded works harder; this one was waiting.

A round trip for every word

Watching the cache server during a single page showed what it was waiting for. One page sent 585 commands to it, and 428 of them were one lookup and its refresh, repeated: a translation table that, for our site, is empty. Every translated string on the page, and a page has hundreds, asked for it again, and each ask held a thread until the answer came back. At three pages a second the threads ran out, and everything queued behind them.

The fix, and a limit

Those lookups are now answered from memory for a minute at a time, fetched once when they expire and cleared on the server that changes a translation. The same page now sends 31 commands instead of 585. We also put a limit in front of page rendering: a fixed number of pages render at once, a short queue waits, and anything beyond that gets an immediate "try again in a moment" instead of slowing every page down.

Measured in production

After the release on 23 September we measured the live site again. The median page renders in 42 milliseconds, down from 249, and the slowest in twenty in 72 milliseconds, down from nearly two seconds. The lookup that ran 214 times per page now runs once.

The lesson we wrote down is plain: a capacity figure is a measurement, not a belief. We had never tested what the public site could take. Now we have, and we will test again before we state one.