PHP-FPM workers that ate the host
RSS grew after every catalog request. The leak was copy-on-write, not a missing unset().
Delivery
Business Cases
Write-ups of past business projects. What the goal was, which decisions we made, and what changed after release.
Memcached for HTML, Redis for the lock
Two stores, one job each. Pages live in Memcached. The rebuild lock lives in Redis.
A locale slab instead of a 2 MB JSON blob
Three locales in one JSON column inflated every listing. A packed slab in Memcached cut the worker heap in half.
Horizon jobs that carried the whole model
Redis filled with serialized Eloquent. The queue was slow because the payload was the catalog.
The Node process that kept every locale
Vue SSR rendered Russian and still allocated English and German message files. The isolate grew with every page.
A microcache that learned every cookie
nginx cached HTML per auth_token. Hit rate died. Public pages do not need that cookie in the key.
A listing that sorted JSON in MySQL
ORDER BY JSON_EXTRACT filled a temp table. Forty rows still cost a filesort over fifty thousand SKUs.
502s from a full listen queue
PHP-FPM was not slow. Docker NAT to port 9000 dropped SYNs when the backlog hit 128.
A scheduler that started the same job four times
withoutOverlapping lived in Memcached. HTML eviction dropped the mutex. Four rebuilds wrote one listing key.
Session keys that never expired
Redis filled with laravel_session blobs. Horizon lost reserved jobs to allkeys-lru.
A bind mount that turned OPcache off
The php service mounted apps/. Every request stat() the tree over virtiofs. Hits in OPcache were noise.
SIGTERM in the middle of a catalog write
compose restart php killed in-flight FCGI. nginx said 502. A half-written Memcached key stayed.
Sleeping PDO handles that filled MySQL
ATTR_PERSISTENT on php-fpm, Horizon, and schedule. max_connections 151. Too many connections at noon.
A date that was yesterday in the browser
Vue SSR printed 14 Jul in UTC. The client printed 13 Jul in Berlin. Hydration failed, then the page rendered twice.
A GIN that never saw the listing query
WHERE title->>'ru' ILIKE scanned fifty thousand JSONB rows. The GIN was for tags. The catalog paid for extract.
Messenger that charged the card twice
The provider returned 200. The worker died before ack. Redis redelivered. Unique middleware did not cover SIGKILL.
A liveness probe that killed PHP during OPcache
The Docker healthcheck became liveness. First request compiled LaraBoom. Kubelet sent SIGKILL. The replica never Ready.
React that hydrated Vue markup
The catalog was Vue SSR. A shared island ran React. Dates disagreed. The page painted twice and shipped the CMS graph.
Edge HTML that ignored the locale cookie
Cloudflare cached the first HTML for the URL. A de visitor got ru markup. Vue hydrated the wrong language, then navigated.