Async Request-Reply
Async request-reply decouples a slow operation from the client that initiated it. The client sends a request, immediately receives an identifier (a correlation ID or job ID), and later retrieves the result by polling a status endpoint or receiving a callback. The request never holds a connection open while the work happens; instead, the work proceeds on a queue and the result is materialized separately. This pattern is how every long-running API works — report generation, video transcode, model training, batch export.
Foundational.
How it works
Async request-reply is the pattern you reach for whenever an operation might take longer than a request timeout. The flow has three phases:
- Submit: the client POSTs to a
/jobsendpoint. The server validates the request, persists it as a job record with a uniquejob_id, enqueues a message onto a work queue, and immediately returns202 Acceptedwith thejob_id. - Poll (or callback): the client periodically GETs
/jobs/{job_id}to check the status —pending,running,succeeded,failed. Alternatively, the client registers acallback_urland the server POSTs to it when the job finishes. - Result retrieval: once the job is
succeeded, the client GETs/jobs/{job_id}/result(or the result URL provided in the status payload) to download the output.
The job_id is the correlation ID — the handle that ties the original request, the queue message, the worker, and the result together. Every log line, every metric, every status check should carry it, because it is the only way to trace an async operation end to end.
There are two flavors of result delivery: polling and callback.
Polling: the client periodically GETs the status endpoint. Simple, firewall-friendly (no inbound connection needed), and easy to implement. The cost is wasted requests (the client polls even when nothing has changed) and latency (the client sees results one polling interval after they happen). Mitigate by allowing the client to set a wait parameter (long polling on the server side) or by using exponential backoff with jitter.
Callback (webhook): the client provides a callback_url at submit time. The server POSTs the result there when the job finishes. Lower latency and zero wasted requests — but the client must operate an HTTP server, handle retries (the callback may fail), verify signatures (the client must prove the callback is from you), and deal with replay attacks. Stripe webhooks, GitHub webhooks, and AWS SNS notifications are all this pattern.
In practice, large systems offer both: polling for clients that cannot receive callbacks, webhooks for clients that can. Some add a third option — WebSocket or Server-Sent Events push — for low-latency real-time updates on top of the same job model.
If a client POSTs /jobs and the request times out before the 202 arrives, the client does not know whether the job was created. The natural reaction is to retry the POST — but if the server is not idempotent, this creates a duplicate job. The fix is the same as everywhere else: accept an Idempotency-Key header, dedupe on it within a TTL window, and return the existing job_id if the key has been seen. Stripe and AWS both do this.
When the result is large (a multi-GB video file, a million-row CSV export), embedding the result in the status response is impractical. The claim-check pattern is the standard solution: the worker uploads the result to object storage (S3) and stores the URL (the 'claim check') in the job record. The status response then contains a result_url pointing to the object. The client downloads from object storage directly, bypassing the API tier — which keeps API response sizes small and lets the client use HTTP range requests, retries, and CDN caching.
This is also how cloud providers handle large async results: AWS operations return an S3 presigned URL where the output will appear once the job completes. The same pattern applies to inputs: if the request payload is large, the client uploads to S3 first and sends only the object key in the POST body.
A production-grade async job system needs explicit lifecycle handling:
- TTL on job records: a succeeded job's status should expire after 7-30 days. Otherwise the jobs table grows forever and slows every query. After expiry, the result is gone; the client must re-submit if they want it again.
- Cancellation: a
POST /jobs/{id}/cancelendpoint that flips the state tocancellingand lets the worker check the flag periodically. Hard cancellation (killing the worker) is rarely clean — cooperative cancellation is the norm. - Progress reporting: workers update a
progressfield (0-100) and anETAestimate. This is what users actually want — not just 'is it done' but 'how far along is it.' - Failure visibility: a
failedstate with an error message and a retry button. Do not return a 500 from the status endpoint — the API call to get status is succeeding; the job itself failed, which is a normal state, not an API error.
A client POSTs to /jobs and the request times out before the 202 arrives. The client retries the POST. What is the correct behavior?
Pick one answer.
Why does the worker store the result URL in the job record instead of embedding the result in the status response?
Pick one answer.
You are choosing between polling and webhooks for an async API. Which constraint most pushes you toward polling?
Pick one answer.
Engineering mental model
Mental model. Think of Async Request-Reply as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Async Request-Reply mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Async Request-Reply, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
message_id = queue.publish({
"type": "async-request-reply",
"key": resource_id
})
# Consumer must be safe to retry.Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Async Request-Reply
Change the variables below and predict what breaks first in Async Request-Reply. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Async Request-Reply, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Async Request-Reply. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Async Request-Reply?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Async Request-Reply, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Async Request-Reply, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Async Request-Reply, separate producer speed from consumer speed. The key design question is what happens when production temporarily exceeds processing capacity: queue it, shed it, slow producers down, or degrade the feature.
Numerical sanity check
A simple queue sanity check: if producers create 8,000 messages/s and consumers process 6,000 messages/s, backlog grows at roughly 2,000 messages/s until the imbalance is corrected.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A downstream service slows down while Async Request-Reply keeps accepting traffic. Would you rather apply backpressure, queue more work, shed load, or degrade? Explain the trade-off.
Pick one answer.
What you gain, what you pay
- +Decouples request lifetime from work duration — no request timeouts on long operations.
- +Survives client disconnects — the job continues; the client can reconnect and check status.
- +Scales naturally — the worker pool is independent of the API tier; backpressure is built in via the queue.
- +Clean cancellation, progress reporting, and retry visibility as first-class concerns.
- +Result can be arbitrarily large — claim-check pattern lets the client pull from object storage directly.
- −More complex client — must handle polling, exponential backoff, and status state machine.
- −Requires server-side state for every job — a job record table that grows and must be TTL-cleaned.
- −Higher latency for short operations — async overhead is not worth it if the work takes 100ms.
- −Webhook variant requires the client to operate an HTTP server and verify signatures.
- −Harder to debug — correlation ID must be propagated across submit, queue, worker, and status endpoints.
How this breaks in production
- Job record table grows unbounded — mitigated by TTL cleanup on terminal states.
- Submit endpoint not idempotent — duplicate jobs on client retry; mitigated by Idempotency-Key.
- Worker dies mid-job — job stuck in `running` forever; mitigated by heartbeat + lease expiry + state recovery.
- Callback URL fails — webhook must retry with exponential backoff; mitigated by dead-letter and visible 'failed delivery' state.
- Status endpoint hot — millions of clients polling every second; mitigated by long polling, SSE push, or longer intervals.
Don't fall into these traps
- •Returning 500 from the status endpoint when the job failed — `failed` is a normal job state, not an API error.
- •Embedding large results in the status response — use claim-check with object storage instead.
- •Not propagating the job_id (correlation ID) through queue, worker, and logs — makes tracing impossible.
- •Forgetting TTL on terminal job records — the jobs table grows forever and slows everything.
- •Polling too aggressively — exponential backoff with jitter, or switch to long polling / SSE.
- •No cancellation endpoint — clients have no way to stop a runaway job.
Real systems using this
How real systems implement this
- Stripe — PaymentIntents are async request-reply: the client creates a PaymentIntent (gets an ID), the server processes asynchronously (3DS, network calls), and the client either polls or receives a webhook when status changes. Idempotency keys on the create call make retries safe.
- AWS long-running operations — Every AWS async API returns an operation ID; clients poll DescribeOperations or register an EventBridge rule to receive a callback. Large outputs (e.g., Athena query results) are written to S3 and the status response references the S3 URI — pure claim-check.
- GitHub Actions / CI systems — Pushing a commit kicks off a workflow run with a unique run ID. The UI polls for status (queued, in_progress, success, failure); webhooks notify on completion. Build artifacts are uploaded to object storage with claim-check URLs in the run record.
- Slack / Discord slash commands — A slash command returns an immediate 200 ACK with a deferred message; the actual response is later POSTed via webhook or via a follow-up API call. This avoids the 3-second Slack timeout for slow operations.
Practice saying it out loud
- Q1Design an async API for generating a large CSV export. What endpoints, what status states, how does the client get the bytes?
- Q2Why must the submit endpoint be idempotent? How would you implement it?
- Q3Compare polling vs. webhooks for async result delivery. When do you choose each?
- Q4How do you prevent the jobs table from growing forever? What about workers that die mid-job?
- Q5How does the claim-check pattern work, and when do you use it?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Claim Check