Redis queue that must survive a second run
The worker can die after the provider call and before the commit. Redis will deliver the job again. The second run has to be empty.
A charge went out twice after I killed the queue container mid-job. The queue in this stack is Redis. Delivery is at least once. A php-fpm worker on the queue container can die after a mail or a webhook left our network and before the local transaction committed. Redis delivers the job again. If the handler is not built for that, you send twice.
The provider had already returned 200. MySQL had not stored the reference. Users saw two charges. The host saw two jobs with the same payload and no unique key.
I do not trust ShouldBeUnique alone. It covers overlap, not a retry after a crash. The fence is a table with a unique key. The key is order id plus step name, never a random uuid. Insert the key before the side effect. If the insert hits the unique index, the work is done and the job returns.
Keep the fence close
The local write sits in a transaction. The external call sits after the key is claimed and before you mark the row complete. The provider response id is stored next to the key. A replay reuses that id instead of charging again.
Retries are bounded. Three attempts with backoff, then failed_jobs and an alert. The job timeout is shorter than the Redis reservation. The backoff is shorter than the queue TTL. A failed payload must be replayable by hand after a fix.
I reproduced it by SIGKILL after the HTTP 200 and before commit. The second worker hit unique, read the stored provider id, and exited. Without the stored id it would have charged again even with the unique key, because the first run never wrote the reference. Claim, call, save id, then complete. Ack or delete the Redis job last.
What I do not put on the queue
Login, CSRF, and the current user endpoint stay synchronous. A queue that must finish before the HTML is sent is a slow request with extra failure modes. Contact form mail can wait. A cache rebuild after a note save can wait. A payment cannot wait without the fence.
The queue container is a second php process in Docker. It shares Redis with cache and with sessions. I keep session keys and job payloads in different prefixes so an eviction of a large list cache does not drop a reserved job. That prefix is part of the design, not an afterthought in the worker log.
Horizon metrics on throughput did not show the double charge. The payment table did. I count rows per order id plus step, and I count provider callbacks. If those numbers disagree, the fence is missing or the id was not saved.
What I took from this
I measure charges and fence rows per order step, not queue throughput. At-least-once Redis will run the handler twice. The second run has to be empty.
I do not trust ShouldBeUnique or a uuid job id. Unique on order id plus step, insert before the HTTP call, store the provider id.
The rule I keep: timeout shorter than the Redis reservation, three tries then failed_jobs, prefixes split from cache. Login stays off the queue.