Web fundamentals, an interactive architecture walkthrough
Message queue: how the architecture works
How a partitioned, replicated log stores messages durably, keeps them ordered per key, and lets consumer groups read at their own pace.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A message queue lets one service hand work or events to others without waiting for them. This design is a replicated, partitioned log: producers append messages to partitions, brokers keep them on disk for a retention period, and consumers pull from the log and record how far they have read. Several consumer groups can read the same messages independently, which makes the log useful for both work queues and event fan-out.
Each partition has a leader broker and followers that copy it, and a write is acknowledged only once the in-sync replicas have it. Ordering is guaranteed within a partition, so related messages share a key. Delivery is at least once: consumers commit their offset after processing and make processing idempotent, and messages that keep failing move to retry and dead-letter topics. This is a simplified learning design, not any specific product's implementation.
Trade-offs and what changes at scale
Waiting for all in-sync replicas makes writes slower but keeps acknowledged messages safe when a broker dies; acknowledging after the leader alone is faster and can lose data. Ordering per partition means the partition count limits consumer parallelism, and adding partitions later changes which partition a key maps to. Classic queues that delete a message once acknowledged, with per-message acks and redelivery, suit task distribution with uneven job lengths better than a log does. At larger scale tiered storage moves old segments to object storage, and consumer lag becomes the main health signal.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
The Order Service hands each event to the Producer Client, which hashes the order id to pick a partition and batches records per partition. Messages with the same key always land in the same partition, which is what keeps them in order.
Before the first send, the Producer Client registers the message schema with the Schema Registry, which rejects changes that would break existing consumers. Messages then carry a small schema id instead of the full schema.
The batch goes to the Leader Broker for that partition with a producer id and sequence numbers. If a network error makes the client resend, the broker sees a repeated sequence and drops the duplicate instead of writing it twice.
The broker appends the batch to the end of the Partition Log, a set of segment files written sequentially. Recent data is read back from the OS Page Cache, so consumers that keep up rarely touch the disk.
Follower Broker 2 and Follower Broker 3 fetch new records from the leader, and the produce request is acknowledged once every in-sync replica has them. Requiring at least two in-sync copies trades some availability for not losing acknowledged messages.
The Controller Quorum tracks which brokers are alive and, if the leader fails, promotes a follower that is in sync. Only in-sync replicas are eligible, so a new leader never lacks acknowledged messages.
Billing Consumer instances join their group through the Group Coordinator, which gives each partition to exactly one member. Adding instances spreads partitions, so the partition count caps how many consumers in a group can work in parallel.
The Consumer Client pulls batches from the leader at its own pace, using long polls when there is nothing new. Pulling lets a slow consumer fall behind without the broker having to buffer or drop messages for it.
The Billing Consumer writes the invoice and the message id to the Billing DB in one transaction, then commits its offset. Delivery is at least once, so the stored id is what turns a redelivered message into a no-op.
Committed positions are stored in the compacted Offsets Topic, one offset per group and partition. A restarted consumer resumes from there, and an operator can rewind the offset to reprocess history.
The Analytics Consumer reads the same partitions in its own group and loads the Analytics Store, without affecting billing. Messages are kept for a retention period rather than deleted when read, so each group has its own independent position.
A message that keeps failing is moved to the Retry Topic so it does not block the partition, and the Retry Worker tries again after a delay. After the last attempt it goes to the Dead-letter Topic for a person to inspect.
Closed segments older than a few hours are copied to Tiered Storage, so long retention does not need large broker disks. Observability watches consumer lag, the number of messages each group is behind.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (21 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Producerslayer
- Order Service: publishes order events
- Producer Client: batching, partitioning, idempotent retries
- Schema Registry: message schemas and compatibility rules
Broker Clusterlayer
each partition: one leader, two followers
- Follower Broker 2: replica in the in-sync set (kafka)
- Follower Broker 3: replica in the in-sync set (kafka)
- Leader Broker: leader for this partition (kafka)
- OS Page Cache: recent segments served from memory
- Partition Log: append-only segment files on disk
- Tiered Storage: old segments offloaded (s3)
Control Planedomain
- Admin API: create topics, set retention
- Controller Quorum: cluster metadata, leader election
Consumerslayer
- Analytics Consumer: consumer group: analytics
- Analytics Store: order facts (clickhouse)
- Billing Consumer: consumer group: billing
- Billing DB: invoices plus processed message ids (postgresql)
- Consumer Client: pulls batches, commits offsets
- Group Coordinator: assigns partitions to group members
- Offsets Topic: committed offset per group and partition (kafka)
Failure Handlingdomain
- Dead-letter Topic: messages that kept failing (kafka)
- Retry Topic: messages waiting for another attempt (kafka)
- Retry Worker: re-processes after a delay
Observability
- Observability: consumer lag, under-replicated partitions
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed cacheHow a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.
- Distributed key-value storeHow a Dynamo-style key-value store places keys on a hash ring, replicates them with tunable quorums, and repairs replicas that fall behind.
- Multi-region active-active setupHow an active-active setup serves users from several regions at once, replicates data asynchronously, and picks a consistency model per data type.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.