Task Queues
A task queue decouples producing work from doing work. Producers push tasks onto a queue; workers pull them off and execute. This enables background jobs (email sending, image processing), smooths bursty load (the queue absorbs spikes), and lets you scale workers independently of web servers. Tools: Celery (Python), Sidekiq (Ruby), SQS (managed), Redis-backed queues.
How it works
A task queue has three components:
- Producer: the code that creates a task. Typically a web server handling a user request — e.g., "user uploaded a photo; enqueue a resize task."
- Queue: the broker (Redis, RabbitMQ, SQS) that holds tasks until a worker picks them up.
- Worker: a process that pulls tasks from the queue and executes them. Workers can be scaled independently of web servers.
The flow: producer enqueues a task (e.g., resize_photo(photo_id=42)). The task is serialized and stored in the queue. A worker pulls the task, deserializes it, executes the function, and acknowledges. If the worker crashes before acknowledging, the queue redelivers the task to another worker (at-least-once delivery).
Key benefits: web response is fast (no waiting for slow work), bursty load is smoothed (queue absorbs spikes), workers scale independently (more workers for image processing, fewer for email), and failures are retried automatically.
Common task queue patterns:
- Fire-and-forget: enqueue a task and don't wait. The web response doesn't depend on the task's result. E.g., "send welcome email."
- Deferred result: enqueue a task and poll for completion. Used when the result is needed but not immediately — e.g., "generate PDF, return URL when ready."
- Scheduled jobs: enqueue a task to run at a specific time or on a schedule (cron). E.g., "send daily digest at 8am."
- Workflow / pipeline: chain tasks where each step's output is the next step's input. E.g., "resize → upload to S3 → notify user." Tools like Celery, Temporal, and Airflow support this.
- Fan-out / fan-in: split a large task into many parallel subtasks, then aggregate results. E.g., "process 1000 images in parallel, then upload the combined result."
- Retry with backoff: failed tasks are retried with exponential backoff, eventually moved to a dead-letter queue if they keep failing.
Each pattern is a building block. Most production systems combine several.
Task queues have delivery semantics:
- At-least-once (most common): a task is delivered at least once. If the worker crashes before acknowledging, it's redelivered. Tasks must be idempotent — running twice must not have side effects (double-charging, double-sending).
- At-most-once: a task is delivered at most once. If the worker crashes, the task is lost. Used only when loss is acceptable (e.g., a metrics sample).
- Exactly-once: theoretically impossible without distributed consensus. Most "exactly-once" systems are at-least-once with idempotency keys making duplicates no-ops.
Make tasks idempotent by including an idempotency key (e.g., the entity ID + operation). The worker checks if it's already processed this key and skips if so. This is the same pattern as idempotent HTTP.
For tasks with side effects (sending email, charging a card), the consumer must dedupe — check a "processed" table or use the provider's idempotency key support (Stripe, SendGrid support this).
Some tasks always fail — a malformed payload, a deleted resource, a bug. Without protection, the queue redelivers them forever, wasting worker capacity. The fix: after N retries, move the task to a dead-letter queue (DLQ). Operators monitor the DLQ; tasks there need manual investigation. Without a DLQ, poison messages can consume a meaningful fraction of worker capacity; with one, they're isolated and the system keeps processing good tasks. Every production task queue needs a DLQ and an alert when it's non-empty.
Task queue tooling landscape:
- Celery (Python): the standard Python task queue. Backed by Redis or RabbitMQ. Mature, feature-rich, complex configuration.
- Sidekiq (Ruby): the Ruby equivalent. Fast, simple, Redis-backed.
- AWS SQS (managed): Amazon's managed queue. No infrastructure to run. Integrates with Lambda (workers = Lambda functions) or ECS. Slightly higher latency than Redis-backed.
- BullMQ (Node.js): Redis-backed, modern, supports scheduled jobs and workflows.
- Temporal (workflow engine): not a simple queue — a full workflow orchestrator with retries, compensations, and state persistence. Used for complex multi-step workflows.
- Airflow (data pipelines): for scheduled data engineering jobs (DAGs). Different niche than general task queues.
Pick based on your stack and needs. For most web apps: Celery/Sidekiq/BullMQ for general tasks, SQS for managed, Temporal for complex workflows. Don't reinvent — these have solved idempotency, retries, DLQs, and scaling.
Your web server handles photo uploads. Each upload triggers a 3-second resize and a 1-second email. Without a task queue, what happens at 100 concurrent uploads?
Pick one answer.
A task to charge a user's card is delivered twice due to a worker crash before ack. How do you prevent double-charging?
Pick one answer.
What's the role of a dead-letter queue (DLQ)?
Pick one answer.
Engineering mental model
Mental model. Think of Task Queues as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Task Queues mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Task Queues, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
message_id = queue.publish({
"type": "task-queues",
"key": resource_id
})
# Consumer must be safe to retry.Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Task Queues
Change the variables below and predict what breaks first in Task Queues. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Task Queues, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Task Queues. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Task Queues?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Task Queues, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Task Queues, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Task Queues, separate producer speed from consumer speed. The key design question is what happens when production temporarily exceeds processing capacity: queue it, shed it, slow producers down, or degrade the feature.
Numerical sanity check
A simple queue sanity check: if producers create 8,000 messages/s and consumers process 6,000 messages/s, backlog grows at roughly 2,000 messages/s until the imbalance is corrected.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A downstream service slows down while Task Queues keeps accepting traffic. Would you rather apply backpressure, queue more work, shed load, or degrade? Explain the trade-off.
Pick one answer.
What you gain, what you pay
- +Web responses are fast — slow work is deferred.
- +Bursty load is smoothed — the queue absorbs spikes.
- +Workers scale independently of web servers.
- +Retries and DLQs handle transient failures and poison messages.
- −Tasks must be idempotent (at-least-once delivery).
- −Adds infrastructure (broker, workers, monitoring).
- −Harder to debug — work is asynchronous and distributed.
- −Latency for the user-visible effect (the email arrives seconds later, not immediately).
How this breaks in production
- Non-idempotent tasks double-execute on redelivery (double charge, double email).
- Poison messages without a DLQ consume worker capacity forever.
- Workers crash silently — tasks not acked, queue grows unboundedly.
- Queue backlog grows faster than workers can process — latency explodes.
Don't fall into these traps
- •Not making tasks idempotent — duplicates on retry.
- •No dead-letter queue — poison messages never go away.
- •Not monitoring queue depth — backlog grows silently until users complain.
- •Holding tasks in memory (no broker) — lost on crash.
Real systems using this
How real systems implement this
- Celery + Redis — Python's standard task queue. Producers enqueue tasks via a Redis broker; workers pull and execute. Supports retries, scheduled jobs, task chains, and a dead-letter pattern via rejected task handling.
- AWS SQS + Lambda — Managed serverless task queue. Producers push to SQS; Lambda functions pull (or are triggered) and process. Scales automatically; supports DLQs, retries, and visibility timeouts.
Practice saying it out loud
- Q1Why use a task queue instead of doing the work synchronously in the request handler?
- Q2Why must task handlers be idempotent? How do you make a non-idempotent operation (charging a card) idempotent?
- Q3What's a dead-letter queue, and why do you need one?
- Q4How do you scale a task queue to handle bursty load?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Message Queues