Get CV
Back to business cases

nginx

502s from a full listen queue

PHP-FPM was not slow. Docker NAT to port 9000 dropped SYNs when the backlog hit 128.

nginx returned 502 in short bursts. PHP-FPM slowlog was empty. The eight workers were busy, not stuck. ss -ltn on the php container showed Recv-Q sitting at 128 on 9000/tcp. New connections got ECONNREFUSED. That is a listen backlog, not an application timeout.

Compose published 9000 through docker-proxy. Each nginx request opened a fresh TCP connection. net.core.somaxconn inside the php image was 128. listen.backlog in the pool file could not grow past that. The NAT layer added latency, so the queue filled on a catalog stampede that Memcached later absorbed.

The problem was a full accept queue on docker-proxy, not slow Laravel. Visitors saw 502 while workers were about to free. I needed a unix socket, keepalive, and somaxconn that matched listen.backlog.

Workers can be free in a second. The accept queue is already full.
Workers can be free in a second. The accept queue is already full.

The socket

nginx and php share a volume with a unix socket. upstream uses keepalive 16. There is no docker-proxy on the hot path. somaxconn is 4096 in the php container. listen.backlog matches it. pm.max_children stayed at eight. That number was never the 502.

Unix socket on the same host. Keepalive on the upstream. No SYN per HTML request.
Unix socket on the same host. Keepalive on the upstream. No SYN per HTML request.
  • error_log with connect() failed still means the backlog, not a fatal in Laravel.
  • TCP 9000 remains for a host PHP that is not in Compose. The public site does not use it.
  • queue and schedule containers talk to Redis, not to this socket.

The 502s stopped on the same traffic that used to fill Recv-Q. PHP-FPM time in the access log did not change. The change was how many handshakes waited on the floor.

What I took from this

Empty slowlog plus 502 is a socket problem until proven otherwise. Recv-Q at the listen limit is the proof.

docker-proxy plus a SYN per request is a queue you will fill on a stampede. Unix socket and keepalive take that queue off the floor.

Raising pm.max_children would have hidden Recv-Q until the host ran out of RAM. The backlog was the real limit.

Back to business cases