Building Reliable Automations With Queues and Background Workers

The request that creates an order should not also render a PDF, email it, resize every image, update an analytics dashboard, and wait for three third party APIs. It should persist the order safely, return a useful response, and hand slow work to a system that can retry it.

Queues and background workers let those operations continue with their own status and retry policy. The application still needs to record what was requested and make the eventual outcome visible.

First decide whether the user must wait

The distinction is about the user facing contract, not about whether a task feels important. A background job can be business critical; it simply needs a durable state and a recovery path.

Queue and Pub/Sub are not interchangeable

A queue distributes work among consumers, with acknowledgement and redelivery behaviour determined by the broker. Publish/subscribe distributes events to subscriptions. Some pub/sub systems retain events for durable subscriptions; others deliver only to currently connected subscribers.

Redis Pub/Sub is good for transient fan out such as real time UI updates. It has at most once delivery: an offline or failing subscriber can miss a message. For work that must survive a restart or be retried, use a durable queue or stream with acknowledgements, such as a Redis Streams based worker system, SQS, RabbitMQ, or another queue that matches the operational requirements.

Validate and persist the requestStore the business change and an outbox event in one database transaction.

Dispatch the outbox eventA dispatcher retries delivery to the durable queue.

Process the jobThe worker uses an idempotency key and records progress.

Persist the result and acknowledgePublish status updates after the durable outcome is known.

The outbox closes the gap between committing a database record and enqueueing its work. A crash between those two operations can otherwise leave an order saved with no PDF job. Delivery may still be repeated, so consumers must remain safe to retry.

Make the worker safe to retry

Retries are expected. Networks fail, provider APIs time out, and deployments interrupt running work. The worker must be designed so that retrying does not create a second charge, email, or document.

Other useful rules: set timeouts for every external call, use backoff with jitter, make the worker observable, and separate “retry later” from “a person must review this.”

A practical stack is still a product choice

For an early stage service, a containerised application on a Hetzner server, deployed through Coolify, with PostgreSQL and a Redis backed durable worker can be enough. For a team using AWS, SQS and managed compute may reduce operational burden. The same design principles apply either way:

• Persist intent before enqueueing work.
• Track job state explicitly: queued, running, completed, failed, and cancelled are much clearer than a spinner.
• Monitor queue age, failure rate, retries, and worker heartbeats.
• Keep logs and metrics together so a failed job can be traced from request to worker.

The check I use before adding asynchronous work

Practical checklist

1. Does the user need the completed result before the request can return?
2. If the process restarts, can this work be recovered without guessing?
3. If it runs twice, can it create a duplicate side effect?
4. Can an operator see why a job failed and decide what to do next?

If the answer to those questions is clear, a queue becomes a reliability tool, not a source of invisible complexity.

For the precise transport guarantees, see Redis Pub/Sub and Redis Streams.