Redis stampede on a three-locale list

After a deploy I flushed notes:list:*. For the next two minutes the public notes list went from 8 ms to 400 ms. Visitors waited on the grid. Redis CPU stayed quiet. MySQL did not.

The list is cached in Redis. The key includes the locale, because English, Russian, and German payloads are different JSON. Twenty php-fpm workers hit a miss at the same time. Each one ran the list query, serialized the collection, and wrote the blob. Three locales, so sixty writes of the same three values. The last writer won. The other fifty-seven were wasted CPU and wasted RAM.

The host felt it as InnoDB load. Users felt it as a slow first paint after every release. Redis only showed the damage later, when LRU started deleting other hot keys that had just been written.

How the memory went

One uncompressed list payload was about 180 KB. Redis maxmemory on this box is 256 MB with allkeys-lru. During the stampede the RSS jumped by roughly 12 MB in a few seconds, which is not the scary number. The scary number is what happened next.

LRU started evicting other hot keys: the current user snippet, a couple of stack catalog fragments, the memcached fallback we keep as a string copy for one endpoint. Those misses pulled more work into PHP. PHP asked Redis again. Redis evicted again. The list keys themselves were large, so they were easy victims once traffic moved on to article pages.

I dumped INFO memory and MEMORY STATS while it was happening. used_memory_peak sat close to maxmemory. evicted_keys climbed in a straight line. Fragmentation was fine. This was not fragmentation. This was too many large strings written at once, then thrown away.

One builder per key

The fix is a lock next to the cache key, not a bigger Redis. For notes:list:ru the lock is notes:list:ru:lock.

SET notes:list:ru:lock 1 NX EX 10
GET notes:list:ru
GET notes:list:ru:stale

The worker that wins SET NX rebuilds the payload, writes the live key with a short TTL, copies it to a stale key with a long TTL, then deletes the lock. Everyone else either waits a few milliseconds and reads the live key, or serves the stale copy if the wait budget is gone.

I keep the stale copy as a separate key on purpose. Overwriting the live key with a longer TTL would hide a bad payload for too long. The stale key is allowed to be wrong for ten minutes. The live key expires in 45 seconds, which is enough to collapse the stampede and short enough that a publish in admin shows up on the next refresh.

What actually sits in RAM

The JSON from LaraBoom still contains all three locales inside each row. Caching the HTTP response per locale means Redis stores the same body field three times, once per key. That is the part I would change next, not the lock.

• Cache the query result once, as a PHP-serialized collection without rendering.
• Pick the locale when writing the HTTP response, not when talking to MySQL.
• gzip the string with gzencode before SET. 180 KB became 28 KB on this list. Redis CPU for decode is cheaper than eviction.

A Hash with fields en, ru, de looks neat and wastes less key overhead. It is also a worse invalidation story. A publish must delete or rewrite the whole hash. Three string keys let me expire one locale without touching the others, which matters when a translator saves only Russian.

I did not put this cache in Memcached. The lock needs SET NX with a TTL, and I want the stale copy next to the live one. Memcached can do add-if-not-exists. It cannot do the rest without extra round trips and uglier expiry.

How I know it holds

The test that matters is not a unit test around a Redis fake. I flush the three list keys, then fire 50 concurrent GETs for /api/notes with Accept-Language left unused and the locale taken from the URL, the same way the Vue SSR entry does it.

Before the lock, MySQL showed about 50 list queries. After the lock, it showed three, one per locale, plus a handful of lock checks. Redis evicted_keys stayed flat. Endpoint p99 dropped back under 20 ms once the stale keys existed.

If the lock is held and both live and stale are empty, the worker must not pile on. I return the empty list for that one request and let the next one rebuild. An empty notes page for 200 ms after a cold start is better than a thundering herd into InnoDB.

What I took from this

After a flush I measure MySQL list queries and Redis evicted_keys, not Redis CPU. Quiet Redis with a hot InnoDB is still a stampede.

I do not trust a bigger maxmemory to absorb sixty identical writes. One SET NX lock per locale key, a 45 second live TTL, and a separate stale copy are the controls that held.

The rule I keep: one builder per cache key, and gzip before SET. If a cold start has no live and no stale, serve empty once. Do not let twenty workers rebuild the same JSON.