Get CV
Back to business cases

Deploy

SIGTERM in the middle of a catalog write

compose restart php killed in-flight FCGI. nginx said 502. A half-written Memcached key stayed.

A deploy is docker compose up -d --build php. The old container gets SIGTERM. PHP-FPM has a few seconds, then SIGKILL. A worker that was writing a listing slab stopped. nginx returned 502. Memcached kept a truncated value. The next GET served broken HTML until the rebuild lock ran again.

opcache_reset() in a hook made the next minute worse. Every surviving worker recompiled LaraBoom at once. CPU spiked. The listing that had just 502 now waited on compile.

The problem was treating a container restart as a drain. Visitors got 502 and then a broken slab. I needed SIGUSR2 inside a live container, a grace period longer than the slowest request, and no shared opcache_reset().

SIGTERM is not a drain. SIGUSR2 is. In-flight FCGI has to finish.
SIGTERM is not a drain. SIGUSR2 is. In-flight FCGI has to finish.

Reload inside the container

The image copies code first. Then kill -USR2 on the php-fpm pid. Old workers finish their request. New workers start with the new files and preload. nginx keeps the unix socket. There is no 502 from a dead FCGI on that path. Memcached writes finish. The rebuild lock in Redis is not left half-held.

A new worker has a new OPcache. A shared reset is a stampede on purpose.
A new worker has a new OPcache. A shared reset is a stampede on purpose.
  • stop_grace_period on the php service is longer than the slowest catalog request we measured.
  • queue and schedule get a new image on their own. A Horizon worker finishes the current job, then dies.
  • Memcached SET of a slab is one value. Partial writes do not happen if the process lives until SET returns.

Deploys still take a minute because of the image build. The public page does not 502 in that minute. That was the actual requirement.

What I took from this

compose restart is not a rolling reload. SIGTERM then SIGKILL cuts FCGI in half. SIGUSR2 is the drain PHP-FPM already has.

opcache_reset() on a live pool is a compile stampede. New workers get a new cache. Old workers should finish and die.

A truncated Memcached value is worse than a 502. Finish SET or do not start it.

Back to business cases