Sequential Convoy
The Sequential Convoy pattern processes a sequence of related messages in strict order per partition, while allowing parallelism across partitions. Messages with the same key (e.g., all events for one order, all transactions for one account) are routed to the same partition and processed serially — guaranteeing that `OrderCreated` is handled before `OrderShipped`. Messages with different keys go to different partitions and process in parallel. The pattern gives you the simplicity of ordered processing within a logical group without sacrificing fleet-wide throughput.
How it works
Sequential Convoy solves the tension between two requirements:
- Order within a related group: messages for the same entity (account, order, conversation) must be processed in the order they were produced.
Deposit $100must be processed beforeWithdraw $50, or the account goes negative. - Parallelism across groups: the system must process many entities concurrently to achieve throughput. Processing one entity at a time is too slow.
The pattern resolves both by partitioning — routing messages with the same key to the same consumer, which processes them serially:
- Each message has a partition key (account_id, order_id, conversation_id).
- A hash of the key determines the partition. Messages with the same key always go to the same partition.
- Within a partition, messages are processed in order — strictly serial.
- Across partitions, processing is parallel — multiple consumers each handle their own partitions concurrently.
Example: 4 partitions, 4 consumers. Account A's messages all hash to partition 0; consumer 0 processes them in order. Account B's messages hash to partition 2; consumer 2 processes them in order. Accounts A and B are processed in parallel by different consumers — but within each account, strict ordering is preserved.
This is the model behind:
- Kafka partitions: messages with the same key go to the same partition; consumers in a consumer group each own partitions.
- AWS SQS FIFO with message groups: messages in the same group are processed in order; different groups are parallel.
- Azure Service Bus sessions: messages in the same session are processed by one consumer at a time, in order.
- Event Hubs partitions: same partition-per-key model as Kafka.
The name comes from the original Microsoft Azure Cloud Design Patterns documentation: a ‘convoy’ of messages traveling together, processed sequentially within their convoy.
What Sequential Convoy guarantees (and doesn't):
Guarantees:
- Per-key ordering: messages with the same key are delivered to the consumer in the order they were produced.
- At-least-once delivery (typical): each message is delivered at least once; may be delivered more than once if a consumer fails mid-processing. Consumers must be idempotent.
- Single consumer per partition at a time: only one consumer processes a given partition; no concurrent processing of the same key. (Across rebalances, a new consumer takes over.)
Does NOT guarantee:
- Cross-key ordering: there's no ordering between partitions. A's message 5 may be processed before B's message 2 even if 5 was produced after 2. If you need cross-key ordering, you need a single partition — which kills parallelism.
- Exactly-once delivery: most implementations are at-least-once. True exactly-once requires transactional consumers (Kafka Transactions API) or idempotent consumers.
- Ordering across consumer rebalances: when a consumer joins or leaves the group, partitions are reassigned. There's a brief window where the new consumer may reprocess messages the old consumer already processed (depending on commit semantics).
Key design decisions:
- Choose the partition key carefully. It must be stable, evenly distributed (no hot keys), and match your ordering requirement. For order processing,
order_id. For account transactions,account_id. For chat,conversation_id. - Beware hot keys. If one key generates 90% of traffic (e.g., a celebrity's account), that partition is a bottleneck. Mitigations: sub-partition by time, hash-sub-key, or special-case hot keys.
- Choose partition count carefully. Too few: limits parallelism. Too many: overhead per partition, and you need enough consumers to handle them all. A common rule: 10-20× the number of consumers, to allow rebalancing.
- Idempotency is required. At-least-once means duplicates; consumers must handle them via deduplication keys, transactional writes, or idempotent operations.
A FIFO queue processes ALL messages in order, globally — one consumer, no parallelism. Sequential Convoy is FIFO within a key but parallel across keys. The two are different points on a spectrum: full FIFO = no parallelism; full parallel = no ordering; Sequential Convoy = ordered per-key, parallel across keys. The choice depends on whether your ordering requirement is global (use FIFO) or per-entity (use Sequential Convoy). Most real systems need per-entity, so Sequential Convoy dominates.
Concrete implementations:
Kafka partitions + consumer groups: messages with the same key go to the same partition; consumers in a group each own a subset of partitions. Within a partition, ordering is guaranteed by Kafka's offset. Consumers commit offsets after processing. Rebalances redistribute partitions across consumers.
AWS SQS FIFO with Message Groups: a FIFO queue supports multiple message groups. Messages in the same group are delivered in order to one consumer at a time. Different groups are processed in parallel by different consumers. Group ID = partition key.
Azure Service Bus Sessions: messages with the same SessionId are processed by one consumer at a time, in order. Sessions are leased to consumers; when the consumer releases or the lease expires, another consumer can take over.
Event Hubs partitions: same model as Kafka. Partitions are the unit of ordering and parallelism.
In all cases, the same concerns apply:
- Consumer failure: if a consumer dies mid-message, the message is redelivered to another consumer. The new consumer may reprocess messages the dead consumer already processed. Idempotency is essential.
- Rebalance: when consumers join/leave, partitions are reassigned. There's a brief window of unavailability or duplicate processing.
- Backpressure: if a partition's consumer is slow, the partition's queue grows. Other partitions aren't affected (good for blast radius).
The pattern is robust and battle-tested. Most large-scale event-driven systems use it as the default model for ordered processing.
You process bank transactions and need: (1) transactions for the same account in order, (2) high throughput across accounts. Which pattern, and what's the partition key?
Pick one answer.
You're using Sequential Convoy and one partition key accounts for 80% of message volume (a hot key — say, a celebrity's account). What's the consequence and the mitigation?
Pick one answer.
Engineering mental model
Mental model. Think of Sequential Convoy as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Sequential Convoy mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Sequential Convoy, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = sequential_convoy(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Sequential Convoy
Change the variables below and predict what breaks first in Sequential Convoy. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Sequential Convoy, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Sequential Convoy. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Sequential Convoy?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Sequential Convoy, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Sequential Convoy, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Sequential Convoy: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Sequential Convoy. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Per-key ordering preserved while enabling parallelism across keys.
- +Scales horizontally by adding partitions and consumers.
- +Simpler than distributed locks or two-phase commit for ordering.
- +Failure isolated per partition — one slow key doesn't block others.
- +Idempotent consumers make at-least-once delivery acceptable.
- −Hot keys cause uneven load — one partition can become a bottleneck.
- −Requires choosing a good partition key — wrong choice means no ordering or hot spots.
- −At-least-once delivery — consumers must be idempotent.
- −Partition count is hard to change later (Kafka requires re-partitioning).
- −Brief duplicate processing during consumer rebalances.
How this breaks in production
- Hot key saturates one partition while others idle — throughput cap.
- Consumer failure mid-message — message is redelivered; duplicate effects if not idempotent.
- Rebalance causes brief duplicate or out-of-order processing during handoff.
- Bad partition key choice — too coarse (one partition for everything) or too fine (no ordering).
- Poison message blocks a partition — bad message can't be processed, blocks subsequent messages in that partition (needs dead-letter and skip).
- Partition count too low — limits parallelism; too high — overhead per partition.
Don't fall into these traps
- •Choosing a partition key that doesn't match the ordering requirement (e.g., transaction_id when you need per-account ordering).
- •Not making consumers idempotent — duplicates cause double effects.
- •Ignoring hot keys until production — load-test with realistic key distribution.
- •Choosing partition count too low to start, then needing to repartition.
- •Not handling poison messages — one bad message blocks a partition forever.
- •Assuming cross-key ordering — there is none; if you need it, use a single partition (and lose parallelism).
Real systems using this
How real systems implement this
- Apache Kafka partitions + consumer groups — Kafka's partitioning is the canonical implementation: messages with the same key go to the same partition (in order); consumers in a group each own partitions; within a partition, ordering is guaranteed by offset. Used by LinkedIn, Netflix, Uber, and most large event-driven systems.
- AWS SQS FIFO with Message Groups — SQS FIFO queues support multiple message groups; messages in the same group are processed in order, while different groups are processed in parallel. Group ID is the partition key. Used for ordered workloads like transaction processing on AWS.
- Azure Service Bus Sessions — Service Bus sessions group messages by SessionId; one consumer at a time processes a session, in order. Sessions are leased to consumers; on lease expiry, another consumer can take over. Used for ordered per-entity processing on Azure.
Practice saying it out loud
- Q1What is the Sequential Convoy pattern, and what problem does it solve?
- Q2How does Kafka implement Sequential Convoy? What guarantees does it provide and not provide?
- Q3You need per-account transaction ordering with high throughput across accounts. Walk me through the design.
- Q4What is a hot key in a partitioned system, and how do you mitigate it?
- Q5Why must consumers in a Sequential Convoy system be idempotent? What happens if they aren't?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Priority Queue