Design Notification System
Design a multi-channel notification system (iOS/Android push, SMS, email, in-app) that fans a single event out to many channels and many recipients. Covers the event bus -> fan-out worker pattern, per-provider rate limiting (APNS/FCM, Twilio, SES), template rendering with localization, idempotency keys to defeat retries, preference & quiet-hours enforcement, and the priority queue that keeps 'your ride is here' ahead of '20% off shoes'.
Foundational.
How it works
What are we designing? A platform that takes a single semantic event from any product service ("order #1234 shipped", "@alice mentioned you", "your OTP is 4471") and routes it to one or more delivery channels per the recipient's preferences. The same event may become an iOS push, an Android push, an SMS, an email, and a banner in the web app — possibly all five, possibly none if the user is in quiet hours.
The defining challenge is heterogeneous fan-out under constraint. Every channel has different throughput limits, latency, cost, and failure semantics. APNS will throttle you, Twilio charges per message, SES will suspend your account if bounce rate exceeds 5%. The system must orchestrate all of this while never sending a duplicate notification and always honoring the user's preferences.
Functional requirements.
- A product service publishes a notification event (recipient, type, payload, channels).
- The system delivers the event through the requested channels (or the user's configured defaults).
- Users can set per-channel preferences (e.g. "SMS only for security alerts, push for everything else, no marketing email ever").
- Users can set quiet hours (no push 22:00-07:00 local).
- The system tracks delivery status (sent, delivered, failed, opened).
- Notifications are deduplicated: a retried producer must not cause two SMSes.
Non-functional requirements.
- End-to-end latency: p99 < 30 s for transactional, < 5 min for marketing.
- Throughput: 10K notifications/sec average, 100K/sec peak (Black Friday, OTP storms).
- At-least-once delivery with idempotent consumers (duplicates are tolerable in rare retry windows but must be the exception).
- No notification storms: a buggy producer must not blast 1M users.
- Provider reputation protection: bounce/complaint rate < 0.5%.
Non-goals. No real-time chat (that's design-chat-system), no in-product marketing campaign orchestration (separate CRM tool), no rich-media push with images (v1).
Capacity estimation.
Volume. 10K notifications/sec avg = ~860M/day. Peak 100K/sec during OTP storms or marketing blasts. ~80% push, 15% email, 5% SMS.
Storage. Each notification record ~1KB (recipient, channel, status, timestamps, payload_hash). 860M/day x 1KB = ~860 GB/day = ~315 TB/year. Hot retention 30 days (~26 TB) in a searchable store; cold archive to S3 after.
Provider rate limits (rough).
- APNS: ~9,000 notifications/sec per cert; we shard across multiple certs/apps.
- FCM: similar, ~10K/sec per project.
- SES: 14 req/sec per region out of sandbox; production limits scale by request.
- Twilio SMS: 10-100 messages/sec per long code, 250/sec per shortcode; throughput varies by country.
Cost (rough order of magnitude). Push ~free (APNS/FCM don't charge). Email ~$0.10 per 1000 messages via SES. SMS varies wildly by country: $0.005-$0.05 per message. A 100M-user marketing SMS campaign is $500K-$5M. This is why SMS is reserved for high-value transactional events.
APIs.
POST /v1/notifications -- send a notification
{ recipient_id, event_type, channels: [...],
payload: { template_id, vars: {...} },
idempotency_key, priority }
GET /v1/notifications/:id/status -- delivery status across channels
POST /v1/preferences/:user_id -- set channel preferences + quiet hours
GET /v1/preferences/:user_idThe producer-supplied idempotency_key is critical. Producers are usually pub/sub consumers that redeliver on failure. Without dedup, a single retry could double every SMS. The notification service hashes (idempotency_key, channel) and rejects duplicates within a 24-hour window using a Redis SET with TTL.
Data model.
Events (Kafka topic, partitioned by recipient_id):
topic: notifications
event: { id, recipient_id, event_type, payload, channels,
priority, idempotency_key, created_at }
npartitions = 64 -- consumer parallelismNotifications (DynamoDB / sharded SQL, key by recipient_id+created_at for fast per-user history):
notifications (
id PK, recipient_id, event_type, channel, status,
payload_hash, idempotency_key, created_at, sent_at, delivered_at
INDEX (recipient_id, created_at)
INDEX (idempotency_key, channel) -- dedup lookup
)Preferences (per-user):
preferences (
user_id PK, channel_prefs JSON,
quiet_hours JSON, locale, timezone
)Per-channel rate-limit state (Redis, per provider):
key: ratelimit:apns:cert_1
type: token bucket (Lua script) -- 9000 tokens/sec, burst 18000Templates (versioned in Git / DB):
templates (id PK, version, channel, locale, body_jinja2, vars_schema)Templates are versioned so a render with v1 of a template is deterministic; rolling out v2 doesn't retroactively change history.
Deep dive: fan-out and per-channel rate limiting.
The whole point of the design is that a single semantic event becomes one or more channel deliveries, each with its own delivery semantics. We do the fan-out as late as possible: the producer says "send to channels [push, email]", the API persists one notification row per channel, and a separate Kafka consumer pool per channel does the actual sending.
Why separate worker pools per channel? Push, email, and SMS have wildly different throughput, latency, and failure characteristics. SMS is slow and expensive; if Twilio is down, we must not stall push delivery. By partitioning consumers per channel, a Twilio outage backpressures only the SMS queue, not the entire system. This is the bulkhead pattern.
Per-channel rate limiting. Each provider publishes a rate limit we must not exceed. We implement a Redis token bucket per provider credential (one per APNS cert, one per FCM project, one per SES region, one per Twilio sender). The Lua script atomically decrements tokens and returns whether the request fits. If it doesn't fit, the worker sleeps with backoff or re-enqueues the message to a delay queue. This guarantees we never blow past provider limits and get throttled.
Priority queue for transactional. Marketing notifications should not crowd out "your ride is here". The push fan-out has a two-tier priority queue: tier 1 (transactional, OTP, security) is drained first; tier 2 (marketing) is drained only when tier 1 is below a depth threshold. We also cap marketing throughput at 10% of provider capacity so a Black Friday campaign never starves OTP delivery.
Bounce and complaint handling. Email providers (SES, SendGrid) deliver webhooks for bounces, complaints, and unsubscribes. We record these against the user and auto-suppress future emails to that address after one hard bounce or one complaint. Failing to do this is the #1 way to get your SES account suspended.
Deep dive: deduplication, preferences, and quiet hours.
Idempotency. Producers retry. Without dedup, an OTP resend would deliver twice and confuse the user. We hash (idempotency_key, channel) and check a Redis SET with 24-hour TTL. On a hit we silently drop the duplicate and increment a dedup_drop metric. This is the same idempotency pattern as Stripe's idempotency keys — producers must generate the key deterministically (e.g. sha256(event_id + channel)).
Preferences. Every notification event has a category (security, transactional, social, marketing). The user's preference record maps category -> enabled channels. Security and transactional categories are non-suppressible (the user cannot opt out of OTP SMS — that's a security requirement). Marketing is fully user-controlled and respects unsubscribe links per CAN-SPAM / GDPR.
Quiet hours. Users can set a local-time window (e.g. 22:00-07:00) during which push notifications are buffered. The buffer is a Redis sorted set keyed by delivery-time; a sweeper releases them at 07:00 local. Urgent notifications (security alerts) bypass quiet hours. We store the user's timezone on the preference record; without it we cannot compute local time.
Template rendering. Each channel and locale has a Jinja2 template. Variables are passed in the event payload; missing variables are rendered as empty rather than crashing. Templates are versioned — a render with v3 of a template is deterministic, so historical notifications can be re-rendered identically. Localization includes not just translation but also date/number formatting (e.g. "12 hours ago" in German vs Japanese).
Bottlenecks and failure modes.
-
Producer storm. A buggy producer emits 1M notifications in 1 minute. Mitigation: per-producer rate limits at the API layer (token bucket per service_id); hard cap on marketing category throughput.
-
APNS/FCM throttle. Pushing 50K/sec will get rate-limited. Mitigation: token-bucket at the provider limit; backpressure to Kafka; APNS supports a
collapse-idto coalesce notifications of the same type for the same device. -
Email bounce spike. A bad list causes 10% bounce rate, SES suspends the account. Mitigation: auto-suppress after one bounce; alert at 0.5% bounce rate; warm up new sender domains gradually.
-
Twilio outage. SMS workers retry with exponential backoff until the message expires (default 5 min). Mitigation: separate bulkhead — push and email still flow. After 5 min, dead-letter the message and surface to ops.
-
Hot recipient. A user mentions 500 people in a single Slack message; 500 fan-out writes for one event. Mitigation: batch by recipient_id; coalesce multiple recent notifications to the same user into one digest.
-
Notification fatigue. A user gets 50 pushes in an hour from the same product and disables notifications app-wide. Mitigation: per-recipient-per-category rate limit ("at most 3 social pushes per hour"); digest mode.
-
Template regressions. A bad template pushes broken HTML emails to 1M users. Mitigation: render to a small canary list (internal team) first; require a manual promote step to go to 100%.
Scaling strategy and trade-offs.
Kafka partitioning. Partition the notifications topic by recipient_id so a given user's notifications serialize through one partition (preserves order, enables per-user dedup). 64 partitions scales to ~100K events/sec with headroom.
Worker autoscaling. Each consumer pool (push, email, SMS) autoscales on consumer lag. Email workers batch send (SES accepts up to 50 destinations per SendEmail call) which reduces cost 50x for marketing.
Multi-region. Deploy the full stack in 2-3 regions. The API is region-affinity (user's region determined by JWT). Kafka MirrorMaker replicates the notification topic cross-region for DR. Provider credentials are per-region where possible (SES region-specific).
Trade-offs made explicit.
- We chose at-least-once delivery with idempotent consumers — gained simplicity and guaranteed delivery, lost true exactly-once (acceptable since dedup covers the common case).
- We chose per-channel worker pools — gained bulkhead isolation, lost some efficiency (a quiet push worker can't help a backed-up SMS worker).
- We chose priority queue for transactional — gained OTP always wins, lost some marketing throughput under load (intentional).
- We chose to fan out late (after Kafka) — gained replayability (events stay in Kafka for 7 days), lost a bit of latency (one extra hop).
- We chose 24h dedup TTL — gained protection against producer retry storms, lost the ability to legitimately send the same notification twice in 24h (acceptable; producers should generate distinct events).
A producer retries the same notification event 3 times in 5 minutes. How do you avoid sending 3 SMSes?
Pick one answer.
Twilio is having an outage. What happens to push notifications and emails?
Pick one answer.
Engineering mental model
Mental model. Think of Design Notification System as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design Notification System mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Design Notification System, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = design_notification_system(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Design Notification System
Change the variables below and predict what breaks first in Design Notification System. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Design Notification System, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Design Notification System. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Design Notification System?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Design Notification System, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Design Notification System, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Design Notification System: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Design Notification System. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Per-channel worker pools isolate provider failures (bulkhead pattern).
- +Idempotency keys with Redis SET dedup make at-least-once delivery practical.
- +Priority queue ensures transactional notifications always beat marketing.
- +Late fan-out (after Kafka) keeps events replayable for 7 days.
- −At-least-once requires every downstream side-effect to be idempotent (extra burden on consumer code).
- −Per-channel worker pools cannot share idle capacity across channels.
- −24h dedup TTL prevents legitimate same-day re-sends (producers must mint distinct events).
- −Template + localization versioning adds operational complexity.
How this breaks in production
- Producer storm floods Kafka — needs per-producer rate limits at the API.
- APNS/FCM throttle from exceeded throughput — needs token-bucket rate limiter per credential.
- SES suspension from bounce spike — needs auto-suppression after one bounce and alert at 0.5%.
- Twilio outage backs up SMS — bulkhead isolation + retry + dead-letter after TTL.
- Hot recipient floods fan-out — needs per-recipient-per-category rate limit + digest mode.
- Template regression sends broken emails — needs canary render before 100% rollout.
Don't fall into these traps
- •Single shared worker pool for all channels — a slow SMS provider stalls push.
- •No idempotency key — producers retry and send duplicate SMSes / push notifications.
- •Treating Kafka exactly-once as covering external side-effects — it only covers offsets.
- •Forgetting to honor quiet hours or locale when rendering templates.
- •Not auto-suppressing bounced email addresses — gets SES account suspended.
- •Sending marketing pushes through the same priority lane as OTPs — OTPs get starved.
Real systems using this
How real systems implement this
- Uber — Internal notification platform ('Floodgate') with per-channel worker pools, idempotency keys, and a priority lane for trip-status pushes above marketing. Documented in Uber Engineering blog posts.
- Slack — Notifications service routes each event to desktop, mobile, and email channels based on per-user preferences and presence state; uses Kafka + per-channel workers.
- Airbnb — Notification service fans out booking events to push, SMS (Twilio), and email (SES); template rendering with locale; per-channel rate limits; documented in Airbnb Medium posts.
Practice saying it out loud
- Q1Design a notification system that sends push, email, and SMS. How do you handle a Twilio outage?
- Q2A producer is retrying the same notification 3 times. How do you avoid duplicate SMS delivery?
- Q3How do you make sure OTP SMSes always beat marketing pushes under load?
- Q4Your email bounce rate just hit 5% and SES is about to suspend you. What do you do?
- Q5How would you add a digest mode that batches 10 social notifications into one email per hour?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Design Rate Limiter