Competing Consumers
Competing Consumers is the parallelism pattern for a work queue: multiple consumer instances pull from the same queue, and the broker ensures each message is handed to exactly one of them. Throughput scales horizontally with the number of consumers up to the broker's partition limit. The catches are ordering (you lose it across consumers), idempotency (a redelivered message may hit a different consumer), and backpressure (more consumers do not help if the bottleneck is downstream).
Foundational.
How it works
A single queue with multiple consumers is the competing consumers pattern. Each message is delivered to exactly one consumer; consumers 'compete' for the next message. This is the default scaling primitive for work queues.
Contrast with pub/sub, where each subscriber gets a copy of every message. Pub/sub fans out; competing consumers divide work. The two patterns compose: a topic can fan out to several subscription queues, each of which is drained by a pool of competing consumers. This is exactly the AWS SNS+SQS fan-out pattern and the Kafka topic+consumer-group model.
The throughput math is straightforward in the happy case: if one consumer processes R messages per second, N consumers process N*R — until you hit the broker's partition limit, the database's write throughput, or some other shared downstream bottleneck.
Adding consumers is the obvious lever, but it has limits:
- Broker partition ceiling. In Kafka, parallelism is bounded by the number of partitions — adding consumers beyond the partition count is wasted, because partitions are assigned one-per-consumer within a group. In SQS, the broker handles fan-out itself, but there is a soft per-queue throughput limit that can be raised on request.
- Downstream saturation. If every consumer writes to the same database, adding consumers shifts the bottleneck from the queue to the database. Queue depth drops; database CPU goes to 100%. You have not solved the problem, just moved it.
- Heterogeneous workers. If workers have different speeds (one is on a slower machine), the slow one becomes the long tail. Use a prefetch limit (RabbitMQ QoS) or maxNumberOfMessages tuning so a slow worker does not hoard a backlog.
The correct diagnostic is to measure where time is being spent: queue wait time vs. processing time vs. downstream latency. Add consumers when queue wait time dominates; fix the bottleneck when processing time dominates.
Once two consumers are pulling from the same queue, you cannot guarantee message A is processed before message B, even if A was enqueued first. Worker A might be slow, or Worker B might pull first. If you need strict per-message ordering, use a FIFO queue with a single partition (single consumer, no parallelism) — or partition by key (so messages with the same key go to the same consumer, preserving per-key ordering).
The bridge between 'competing consumers' and 'preserved per-key ordering' is partitioning by key. The broker hashes each message's partition_key onto one of P partitions. Each partition is consumed by exactly one consumer within a consumer group. So:
- Across partitions: messages are processed in parallel by different consumers; no ordering guarantees.
- Within a partition: messages are processed in FIFO order by a single consumer.
This is the Kafka and SQS-FIFO model. The trade-off is that a single hot partition (e.g., all messages for one popular user) cannot be parallelized — you are limited by the single consumer assigned to it. Hot keys are the silent killer of partitioned throughput; consistent hashing and a high enough partition count are the standard mitigations.
When a consumer crashes or rejoins, the broker rebalances partitions across the live consumers. During rebalance, in-flight messages may be redelivered to the new owner of the partition — which is why consumers must be idempotent even within a single partition.
Prefetch / QoS controls how many messages a consumer can hold in-flight before acking. A prefetch of 1 means each consumer processes one message, acks, then pulls the next — slow, but fair. A prefetch of 100 means the consumer can buffer up to 100 messages, which improves throughput by hiding broker latency but means one slow consumer can hoard 100 messages that could have gone to faster consumers.
The rule of thumb: set prefetch high when all consumers are roughly equally fast; set it low (1-5) when workers are heterogeneous. For latency-sensitive work, prefetch 1 with long polling. For throughput-oriented work, prefetch 50-100 with batch acks.
You have 12 Kafka partitions and 20 consumer instances in one consumer group. How many consumers actually do work?
Pick one answer.
Your queue depth is growing. You add consumers, but depth keeps growing. What is the most likely diagnosis?
Pick one answer.
You need strict per-order ordering for messages within the same order_id, but want parallelism across orders. Which design do you choose?
Pick one answer.
Engineering mental model
Mental model. Think of Competing Consumers as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Competing Consumers mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Competing Consumers, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = competing_consumers(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Competing Consumers
Change the variables below and predict what breaks first in Competing Consumers. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Competing Consumers, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Competing Consumers. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Competing Consumers?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Competing Consumers, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Competing Consumers, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Competing Consumers, separate producer speed from consumer speed. The key design question is what happens when production temporarily exceeds processing capacity: queue it, shed it, slow producers down, or degrade the feature.
Numerical sanity check
A simple queue sanity check: if producers create 8,000 messages/s and consumers process 6,000 messages/s, backlog grows at roughly 2,000 messages/s until the imbalance is corrected.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A downstream service slows down while Competing Consumers keeps accepting traffic. Would you rather apply backpressure, queue more work, shed load, or degrade? Explain the trade-off.
Pick one answer.
What you gain, what you pay
- +Throughput scales horizontally with consumer count (until the broker or downstream saturates).
- +Natural fault tolerance — if a consumer dies, the broker redelivers to another.
- +Load balancing is implicit — the broker hands work to whichever consumer is idle.
- +Composes cleanly with pub/sub fan-out (topic -> multiple subscription queues, each with competing consumers).
- −Strict FIFO is lost across consumers; only per-partition ordering remains.
- −Idempotency becomes mandatory — a redelivered message may hit a different consumer than the original.
- −Hot keys cannot be parallelized — a single partition key with high volume bottlenecks one consumer.
- −Bounded by broker partitions (Kafka) or broker-level throughput caps (SQS).
How this breaks in production
- Adding consumers does not help — downstream saturation (usually the database) is the real bottleneck.
- Hot partition — a single high-volume key saturates one consumer while others idle.
- Prefetch too high — slow consumers hoard messages that faster consumers could process.
- Rebalance storms — consumers churning (deploy, autoscale) cause constant partition reassignment, with redelivery.
- Ordering violation — assuming global FIFO when only per-partition FIFO is guaranteed.
Don't fall into these traps
- •Adding more consumers than partitions (Kafka) — wasted instances do nothing.
- •Assuming cross-consumer ordering — only per-partition/per-key ordering is preserved.
- •Forgetting that consumers must be idempotent — partition rebalances and ack losses cause redelivery.
- •Treating queue depth as the only signal — also measure per-message processing time to find downstream bottlenecks.
- •Setting prefetch too high on heterogeneous workers — slow workers hoard messages.
- •Not handling consumer-group rebalances gracefully — long-running consumers should commit offsets incrementally.
Real systems using this
How real systems implement this
- Apache Kafka consumer groups — Each consumer group is a competing-consumers pool across partitions. Within a group, each partition is assigned to exactly one consumer. Adding consumers beyond partition count is idle. Rebalances on consumer join/leave can cause short redelivery windows.
- AWS SQS + Lambda / ECS workers — Multiple Lambda functions or ECS tasks pull from the same SQS queue. The broker handles delivery arbitration; with batch sizes of 10 and long polling, throughput scales with concurrency up to the queue's account-level burst limit. DLQs handle poison messages.
- Sidekiq / Celery — Redis-backed job queues with N worker processes competing for jobs. Concurrency is per-process; horizontal scaling is per-host. Prefetch is typically 1 (job is popped and processed before the next is fetched).
- GitHub Actions runners — Self-hosted or hosted runners compete for queued workflow jobs. The queue dispatches one job per idle runner; throughput scales with runner count, bounded by the concurrency limit on the repo/org.
Practice saying it out loud
- Q1Your queue depth is growing. You add consumers, but depth keeps growing. Walk me through the diagnosis.
- Q2How does Kafka partitioning interact with consumer groups? What happens when you add a consumer beyond the partition count?
- Q3You need strict per-order ordering for messages but want cross-order parallelism. How do you design this?
- Q4What is prefetch / QoS, and how do you tune it for homogeneous vs. heterogeneous workers?
- Q5When does adding competing consumers make the situation worse instead of better?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Message Queues