Memcached for HTML, Redis for the lock

The listing cache expired for everyone at once. Memcached dropped the key, two hundred requests went to MySQL, and the catalog looked like a traffic spike. It was a clock spike. Every TTL had been set to 60 seconds from the same write.

The stack already had two memory stores. CACHE_STORE pointed at Memcached. QUEUE_CONNECTION and SESSION_DRIVER pointed at Redis. I stopped treating them as interchangeable and gave each one a job it is good at.

The problem was a thundering herd on a synchronized TTL. Shop visitors waited on MySQL. I needed one rebuild, a stale twin for everyone else, and a lock that HTML eviction could not drop.

Why not one store

Memcached is a slab allocator. It is fast at get/set of blobs and bad at locks. A GET that misses cannot promise that only one caller rebuilds. Redis can. SET key NX EX 5 is a lock with a TTL, and the same Redis instance already runs Horizon.

Putting HTML into Redis would have worked. It would also have mixed page blobs with queue payloads and session hashes in one eviction policy. When Redis hit maxmemory, a catalog page and a reserved job could leave together. That mix is how you get a stampede and a stuck queue on the same afternoon.

The lock

On a miss the worker tries SET lock:rebuild:{key} NX EX 5 in Redis. If it gets the lock it reads MySQL, writes Memcached, deletes the lock. If it does not, it serves the previous HTML from a twin key that never expires until the next successful write. Stale for two seconds beats a stampede.

TTLs on listing keys are jittered. The write adds a random offset of a few seconds so the whole grid does not expire on one clock tick. The twin key is the last good HTML. It is not a second source of truth. MySQL still is.

nginx in front

Anonymous hits also go through a two-second microcache in nginx. The cache key is scheme, host, uri, and locale. It ignores query strings that do not change the page. Requests with the auth_token cookie bypass that cache, so a signed-in user never receives a public fragment.

Hit rate on the listing moved from single digits to most requests. MySQL CPU on that host fell to a flat line between writes. The code for this is one helper. New cached reads get the lock and the twin key without a new design.

What I took from this

Two memory stores are not redundancy. They are two jobs. Blobs in Memcached. Locks and queues in Redis. One eviction policy for both is how a stampede and a stuck Horizon show up together.

A synchronized TTL is a scheduled outage. Jitter and a lock beat a longer TTL that still expires for everyone at once.

Stale HTML for two seconds is a product choice I will take over two hundred identical SELECTs.