Messaging & social, an interactive architecture walkthrough
Notification system at scale: how the architecture works
How a notification platform validates requests, applies user preferences, queues by priority, and delivers push, SMS and email with retries and fallbacks.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A notification platform sits between product teams and delivery providers. Requests are authenticated, deduplicated, checked against user preferences and rendered from templates, then placed on per-channel, per-priority queues so a marketing burst does not queue behind one-time passwords (separate topics and worker pools; shared brokers and provider rate limits can still add delay). Workers call the providers, send failed or throttled messages to delay topics for a later retry, and record status: provider webhooks for SMS and email, and accepted-or-rejected results for push.
Trade-offs and what changes at scale
Queues with retries give at-least-once delivery, so duplicates are possible and the idempotency check is what keeps them rare; exactly-once is not on offer across third-party providers. Separate topics and worker pools per channel and priority isolate failures and let each scale on its own, at the price of more things to run. Kafka has no per-message delay, so retries use delay-tier topics or a scheduler. Large broadcasts are split into batches so they do not starve transactional traffic, and each provider's rate limit becomes the real ceiling, so email (and often SMS) get a second provider; APNs and FCM cannot substitute for each other. Every notification carries an expiry, so time-sensitive messages such as OTPs are dropped rather than retried late.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
A product service calls the notification API. The gateway is the single entry point, so every team is handled uniformly.
The gateway authenticates the calling team and validates the payload, rejecting bad requests before they cost any downstream work.
The notification service checks the idempotency key against a durable store and deduplicates, so a retry carrying the same idempotency key is recognised and not sent again while the key is retained.
User preferences, opt-outs, quiet hours and per-user frequency caps are checked before anything is queued; suppressed channels are dropped. Critical messages like OTPs bypass marketing opt-outs and quiet hours (legal opt-outs still apply).
The template is picked for channel and locale and rendered with parameters. Push tokens come from the device registry, where the apps register them through the API.
The message is enqueued on per-channel, per-priority topics so a marketing burst does not queue behind OTPs. Queues absorb spikes.
A worker consumes the message and checks provider-side throttles in Redis. Excess is deferred to a delay topic and retried later instead of hammering the provider; per-user caps were already applied before queueing.
The push pool consumes its topic and calls APNs or FCM. Pools scale independently per channel. A successful call only means the provider accepted the message, not that it was delivered.
The SMS and email pools do the same with their own topics: SMS to Twilio, email to SendGrid. Again, a successful call only means the provider accepted the message.
On a transient provider failure, the worker publishes the message to a retry topic with a growing delay (Kafka has no per-message delay, so delay tiers are used, and the delay is best effort: a retry runs at least that late, possibly later); a relay re-enqueues it when due, back onto its original priority topic. Every notification carries an expiry, so critical messages like OTPs skip the long tiers and are dropped when expired. Email fails over from SendGrid to SES on definite errors or timeouts, accepting a small risk of a duplicate email if the first provider accepted the message but the reply was lost. Messages that exhaust their retries land in a dead-letter queue, where a replay tool can list, purge and merge them back into the first retry topic.
Providers report back differently. SMS and email providers call webhooks with delivered, bounced or failed events, and the workers also record the immediate accepted-or-rejected result. APNs and FCM do not call back: they only answer the send request, so push status is 'accepted by provider'. Real delivery or opens appear only if the app or SDK reports them; FCM offers opt-in aggregated Android delivery data and per-message events through BigQuery export, while APNs only reports acceptance. Tokens are pruned only on definitive 'token is dead' answers (APNs 410 Unregistered or BadDeviceToken, FCM UNREGISTERED); FCM INVALID_ARGUMENT, quota and server errors are treated as retry or payload problems. The tracker stores all of it for reporting.
In-app messages skip external providers and are written to the inbox store that the app reads.
The scheduler keeps its jobs in a job store and fires delayed, recurring and batch work through the same gateway, so all policies apply equally. Large broadcasts are expanded into batches by a campaign service first, so they do not starve transactional traffic.
Observability gathers metrics from services, workers, DLQs and tracker: queue lag, send latency, failure rates, alerts.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (31 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
API Layerlayer
- API Gateway: Authn, validation, quotas
- Auth Service: API keys / OAuth per team
Callersboundary
- Admin Console: manage templates and preferences
- Campaign Service: expands a broadcast into batches
- Internal Services: Product teams' backends
- Mobile Apps: register push tokens
- Scheduler: Delayed, recurring, batch campaigns
- Scheduler Job Store: due and recurring jobs
Notification Coredomain
- Device Token Registry: Push tokens per user/device
- Idempotency Store: durable dedupe keys with expiry
- In-App Inbox: Stores and serves in-app messages
- Notification Service: Orchestrates and applies policy
- Preferences DB: User preferences and opt-outs
- Rate Limiter: User frequency caps, provider throttles (redis)
- Template Store: Templates and locales
Queue Layerlayer
- Dead-Letter Queues: Exhausted retries, per channel (kafka)
- Email Topics: critical / marketing topics (kafka)
- Push Topics: critical / marketing topics (kafka)
- Retry / Delay Topics: delay tiers (e.g. 1 min, 5 min, 30 min) (kafka)
- SMS Topics: critical / marketing topics (kafka)
Channel Workerslayer
- Email Workers: Send, retry via delay topics
- Push Workers: Send, retry via delay topics
- Retry Relay: waits out each delay, then re-enqueues
- SMS Workers: Send, retry via delay topics
Third-Party Providersboundary
- Amazon SES: Email fallback
- APNs: iOS push
- FCM: Android/web push
- SendGrid: Email primary (sendgrid)
- Twilio: SMS primary (twilio)
Delivery Trackingdomain
- Delivery State DB: State and analytics events
- DLQ Replay Tool: list, purge, merge back for replay
- Status Tracker: Provider webhooks and send results
Observability
- Observability: Metrics, logs, alerts
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- WhatsApp-style chat appHow a chat system holds millions of persistent connections, routes end-to-end encrypted messages, and falls back to push when users are offline.
- Chat application (1:1 and group)How a chat app keeps a WebSocket per online user, orders and stores messages per channel, and lets reconnecting clients catch up from history.
- Social news feed (Twitter-style timeline)How a social feed pushes posts into follower timelines, handles accounts with millions of followers by merging at read time, and ranks the result.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.