A scheduler that started the same job four times
withoutOverlapping lived in Memcached. HTML eviction dropped the mutex. Four rebuilds wrote one listing key.
The schedule container runs php artisan schedule:run every minute. catalog:rebuild was everyMinute() with withoutOverlapping(). The job needed four minutes on a dirty set of a few thousand ids. Four processes wrote the same Memcached keys. Redis catalog:dirty never went empty because each run added ids the previous run had not finished.
withoutOverlapping() stores a lock in the default cache. CACHE_STORE was Memcached. A listing slab and the mutex shared a slab class. When the catalog filled the cache, the lock key left. The next minute saw no mutex and started again.
The problem was a mutex in the store that throws HTML away on purpose. Four rebuilds fought one listing key. The dirty set never drained. I needed the lock next to Horizon, in Redis, with a TTL longer than the slowest rebuild.
The lock store
The scheduler lock moved to Redis. SET schedule:rebuild NX EX 600. Horizon already lives there. HTML eviction in Memcached cannot drop that key. The job still reads dirty ids from a Redis set, builds slabs, and writes Memcached. Only one builder runs.
- onOneServer() uses the same Redis, so two schedule replicas do not both start.
- The timeout on the lock is longer than the slowest rebuild we have measured.
- A missed minute is fine. A doubled rebuild is not.
The dirty set now drains. Memcached sees one writer. The schedule container CPU went back to idle between ticks. The bug was not the cron. The bug was putting a lock in the store that throws pages away on purpose.
What I took from this
withoutOverlapping() is only as good as the store behind CACHE_STORE. Memcached is not a lock service.
A mutex TTL shorter than the job is a second start. EX 600 is boring and correct.
I will miss a minute before I run two rebuilds. Overlap costs more than delay.