A background job is reliable when the product can tolerate it running late, more than once, or after a process crash—without silently losing the intended work. “We call a queue after saving” is not that guarantee.
Make the handoff durable
When a request changes durable state and requires follow-up work, store both the state change and a pending event in the same database transaction. A separate worker reads that outbox, delivers the job, and records delivery attempts.
flowchart LR
accTitle: Durable background work handoff
accDescr: An API transaction stores product state and an outbox event. A worker delivers the event to a queue and a job consumer performs idempotent work with monitoring and replay support.
A["API transaction"] --> D[("Product state")]
A --> O[("Outbox event")]
O --> W["Delivery worker"]
W --> Q["Queue"]
Q --> C["Idempotent consumer"]
C --> M["Logs, metrics, alerts"]Without this pattern, a server can commit the request and crash before it publishes the message. The user sees success; the downstream work never happens.
Assume at-least-once delivery
Most practical queues may deliver more than once. Make consumers idempotent using a durable operation key, a unique constraint, or an explicit processed-event record. Do not try to promise “exactly once” by assuming a network call cannot time out after the remote system accepted it.
Retrying needs a policy
Classify errors. Retry temporary provider failures with exponential backoff and a bounded attempt count. Do not retry invalid input forever. Send exhausted jobs to a visible failure state, alert an owner, and provide a safe replay path after the root cause is fixed.
Observe the work, not just the queue
Track pending outbox age, job latency, success rate, retry count, and dead-letter count. Include a correlation ID from the initiating request in each log and downstream call. The operational question should be answerable: “Did this customer action finish, and if not, where is it stuck?”
Reliable job design is a core part of production readiness, not infrastructure polish for later.