Claim Check

Message queues are optimized for small, fast messages. They're meant to handle thousands of small messages per second, with low latency and predictable throughput. They are not designed to handle large payloads:

• Size limits: SQS hard limit 256KB, Kafka default 1MB (configurable to higher, but discouraged), Azure Service Bus 256KB (Standard) / 100MB (Premium).
• Broker memory pressure: large messages consume broker memory and slow down all messages in the queue.
• Storage cost: most brokers charge for storage; large messages cost more.
• Throughput impact: a 100 MB message blocks the queue for the duration it takes to write/read.

The Claim Check pattern solves this:

1. The producer has a large payload (e.g., a 100 MB video to be transcoded).
2. The producer uploads the payload to object storage (S3, Azure Blob, GCS), getting back a URL.
3. The producer publishes a small message to the queue, containing the URL (the claim check) and metadata (size, type, content hash).
4. The consumer receives the small message quickly, then fetches the large payload from object storage using the URL.
5. After processing, the consumer may delete the payload from storage (or a TTL handles it).

The analogy is the coat check at a theater: you give the attendant your coat (large), they give you a small numbered token (the claim check). You don't carry the coat around — you carry the token, and retrieve the coat when you need it. The queue is the token distribution system; the coat check room is the object store.

Crucially, the claim check itself is small (a few hundred bytes: URL + metadata), so the queue stays fast even when the payloads are huge.

Design decisions for the Claim Check pattern:

Where to store the payload?

• Object storage (S3, Azure Blob, GCS) — most common; cheap, durable, scalable, supports TTLs and lifecycle rules.
• A database BLOB column — for smaller large payloads (1-50 MB) where you want transactional consistency with the message.
• A dedicated blob store (Redis, MinIO) — for very short-lived payloads (seconds to minutes).

How long to keep the payload?

• Until the consumer confirms processing — consumer deletes the payload after successful processing. Requires the consumer to have delete permissions.
• TTL-based — set a TTL on the payload; if the consumer never processes it, the payload auto-expires. Decouples payload lifecycle from consumer success.
• Indefinitely — for payloads that may need to be re-processed (replays, bug fixes). Storage costs add up.

What metadata to include in the claim check?

• URL/key — required, the pointer to the payload.
• Size — so the consumer knows what it's fetching.
• Content type — so the consumer knows how to parse.
• Hash (SHA-256) — for integrity verification; the consumer can detect corruption.
• Source/timestamp — for debugging and tracing.
• Compression info — if the payload was compressed before upload.

Access control?

• Public read — if the payload is non-sensitive; consumer fetches via plain URL.
• Presigned URL — producer mints a presigned URL (Valet Key) for the consumer; tight scope, short expiry.
• IAM role-based — consumer has its own credentials to read from the bucket.

Cleanup?

• A common bug: payloads accumulate in storage forever because nobody deletes them. Always have a cleanup strategy — TTL, post-processing delete, or a periodic GC job.

When to use Claim Check:

• Payload exceeds broker's size limit (e.g., SQS's 256KB). Required, not optional.
• Payload is large (>1MB) and would slow the broker or other messages. Even if technically allowed, large messages hurt broker performance.
• Multiple consumers need to fetch the same payload. Store once, queue references multiple times.
• The payload is large but rarely consumed. E.g., a daily report that most consumers don't read; let them fetch on demand.
• The payload needs to outlive the message. Keep the message small and short-lived; the payload lives in storage until explicitly deleted.

When NOT to use Claim Check:

• Payloads are small (<100KB). No benefit; the indirection adds latency.
• The consumer needs strict ordering with the payload. Claim checks add an extra hop; if you need atomicity, a single small message is better.
• You're already in a system that handles large messages well (e.g., Kafka configured for large messages, with the operational cost accepted).
• Latency is critical — the extra hop to object storage adds 50-200ms.

A common variation: store the payload in object storage, then queue a small message with the URL and a few essential fields. The consumer can decide whether to fetch the full payload based on the metadata — skipping fetches for uninteresting messages entirely. This is the ‘envelope with metadata + claim check’ pattern, common in event-driven architectures.

Mental model. Think of Claim Check as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Claim Check mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Claim Check, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.