Background Jobs
A background job is a unit of work performed outside the request/response cycle. Instead of doing slow, expensive, or non-critical work synchronously while the user waits, the application enqueues a job (with a message broker or database row) and returns immediately. Workers pick up the job asynchronously and execute it. Patterns include fire-and-forget, scheduled jobs (cron), deferred work, and retries on failure.
How it works
The problem:
A user signs up. Your handler:
- Inserts a row in the database — 5ms.
- Sends a welcome email via SendGrid — 800ms.
- Generates an onboarding PDF — 1.5s.
- Calls an analytics service — 300ms.
Total: ~2.6 seconds of waiting. The user stares at a spinner for nearly three seconds for a signup that should be instant. Worse: if SendGrid has an outage, your signup endpoint times out — users cannot even create an account, because your third-party email provider is down.
The lesson is general: synchronous work should be the minimum needed to acknowledge the user's intent. Everything else belongs in the background. The signup handler should insert the user row, return 200, and let background workers handle the email, PDF, and analytics. The user gets a fast response; slow or failing dependencies don't break the core flow.
Patterns of background work:
- Fire-and-forget — enqueue a job and never check back. Sending a welcome email, writing a metric, posting to a webhook. The job runs once; failures are logged and retried but the caller does not wait.
- Deferred work — schedule a job to run at or after a specific time. 'Send the onboarding email 24 hours after signup.' 'Archive this draft in 7 days if unused.' Implemented with delayed-queue semantics (Sidekiq scheduled jobs, SQS delay seconds, Celery ETA).
- Scheduled / cron jobs — run periodically: nightly rollups, expired-session cleanup, daily reports. Triggered by a cron scheduler that enqueues jobs at fixed times.
- Long-running jobs — video transcoding, ML training, large report generation. Often split into chunks, with progress tracked in the database and surfaced to the user via polling or WebSocket.
- Fan-out / fan-in — one trigger enqueues many jobs (process every frame of a video); when all finish, a final job aggregates results.
Underlying all of these is a job queue: a broker (Redis, RabbitMQ, SQS, Kafka) that holds the jobs and a worker pool that consumes them.
When to move work out of the request path:
A useful decision rule: if a piece of work is not strictly required to produce the response the user is waiting for, defer it.
Concrete signals:
- The work takes more than ~50ms and the user doesn't need to see the result immediately.
- The work depends on a third party whose availability you don't control (payments, email, analytics, AI).
- The work is non-critical: if it fails, the user's primary intent should still succeed.
- The work is periodic (cron), not triggered by a request at all.
- The work is expensive enough that you want to retry it on failure (image processing, document generation).
Conversely, keep work synchronous when:
- The user needs the result of the work to act (a checkout must authorize the charge before confirming the order).
- The work is cheap and reliable (a database read, a cache check).
- The cost of async (broker, worker, idempotency) outweighs the latency savings.
Idempotency is the silent requirement for any background job. Because workers retry on failure, a job may run more than once. If the job sends a welcome email, double-sending on retry is annoying. If it charges a credit card, double-charging is a regulatory incident. Every background job must be safe to execute multiple times — typically by tracking an idempotency key in the database.
A job that always fails (e.g., a malformed payload, a deleted user, a code bug) will be retried until it hits the retry limit, then dropped into a dead-letter queue. But while it is being retried, it ties up a worker and consumes broker capacity. A flood of poison jobs can starve healthy jobs behind them. Mitigations: strict retry limits (e.g., 5 attempts with exponential backoff), per-job timeouts, dead-letter queues for inspection, and alerts when the DLQ is non-empty. Always bound retries — infinite retries are a self-inflicted DoS.
Your signup handler sends a welcome email synchronously via SendGrid. SendGrid has a 10-minute outage. What is the user-visible impact, and how would background jobs change it?
Pick one answer.
A background job charges a customer's credit card. The job succeeds, but a network blip prevents the worker from receiving the success response. The broker redelivers the job. What happens, and how should you prevent the bad outcome?
Pick one answer.
Which of the following is the strongest signal that work belongs in the request path (synchronous), not the background?
Pick one answer.
Engineering mental model
Mental model. Think of Background Jobs as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Background Jobs mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Background Jobs, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = background_jobs(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Background Jobs
Change the variables below and predict what breaks first in Background Jobs. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Background Jobs, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Background Jobs. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Background Jobs?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Background Jobs, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Background Jobs, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Background Jobs, separate producer speed from consumer speed. The key design question is what happens when production temporarily exceeds processing capacity: queue it, shed it, slow producers down, or degrade the feature.
Numerical sanity check
A simple queue sanity check: if producers create 8,000 messages/s and consumers process 6,000 messages/s, backlog grows at roughly 2,000 messages/s until the imbalance is corrected.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A downstream service slows down while Background Jobs keeps accepting traffic. Would you rather apply backpressure, queue more work, shed load, or degrade? Explain the trade-off.
Pick one answer.
What you gain, what you pay
- +Fast user-facing requests — slow work moves off the critical path.
- +Resilience to third-party failures — jobs retry on backoff instead of failing the request.
- +Better throughput — workers scale independently of web servers; you can size each for its workload.
- +Scheduled and deferred work (cron, ETA) becomes first-class.
- +Failure isolation — a buggy job doesn't take down the API.
- −Adds infrastructure: a broker (Redis, SQS, RabbitMQ, Kafka) and a worker pool.
- −Jobs may run more than once — every job must be idempotent, which is real design work.
- −No longer real-time — the user waits for the work to be picked up. UX must reflect this (polling, webhooks, or progress UI).
- −Debugging is harder — failures are no longer in the request's stack trace; you need job IDs and logs.
- −Operational surface: dead-letter queues, retry limits, monitoring for stuck jobs, alerting on backlog growth.
How this breaks in production
- Poison-pill jobs that always fail and are retried forever, starving healthy jobs behind them (mitigate with retry limits + DLQ).
- Non-idempotent jobs double-executing on redelivery (mitigate with idempotency keys).
- Backlog growth when workers can't keep up with enqueue rate (mitigate with autoscaling and alerting on queue depth).
- Job loss when a worker crashes mid-execution (mitigate with at-least-once delivery and idempotency — at-most-once is rare and dangerous).
- Long-running jobs blocking workers (mitigate with per-job timeouts and splitting into chunks).
Don't fall into these traps
- •Doing slow third-party calls synchronously in the request path — one outage takes down your API.
- •Background jobs that aren't idempotent — redelivery causes duplicate side effects.
- •No retry limit — poison jobs retry forever, exhausting worker capacity.
- •No dead-letter queue — failed jobs vanish silently with no way to inspect them.
- •No monitoring of queue depth — a slow leak in workers goes unnoticed until users complain.
- •Treating 'fire-and-forget' as truly fireable — forgetting that the worker still needs to log, retry, and surface failures.
Real systems using this
How real systems implement this
- Sidekiq (Ruby) — Redis-backed background job framework. Workers consume jobs from Redis queues; jobs support retries with exponential backoff, scheduled execution, and dead-job queues. Widely used in Rails apps to move email sends, PDF generation, and third-party API calls off the request path.
- AWS SQS + Lambda — SQS queues hold messages; Lambda functions consume them with automatic scaling and built-in retries. Failed messages move to a dead-letter queue after a configurable number of attempts. The default serverless background-job stack on AWS.
- Stripe webhook delivery — Stripe does not process your webhook handler synchronously when a payment event occurs — it enqueues a webhook delivery and retries on failure with exponential backoff over up to 3 days. Your handler just needs to acknowledge receipt quickly and do its work idempotently.
Practice saying it out loud
- Q1Walk through signup, password reset, and order checkout. For each, what work belongs in the request path and what belongs in the background?
- Q2Your background job is sometimes executed twice, causing duplicate side effects. How do you prevent this, and what broker guarantees are you relying on?
- Q3A worker pool's queue depth is growing. What do you check, and how do you remediate?
- Q4How would you design a background job that charges a customer's card, sends a receipt, and updates inventory — safely across failures?
- Q5When is it correct to do work synchronously even though it's slow?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Task Queues