SaaS & data, an interactive architecture walkthrough
Metrics and monitoring system: how the architecture works
How a monitoring system scrapes metrics from every service, alerts through grouped and routed notifications, and keeps years of data in object storage.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A metrics and monitoring system collects numeric measurements from every service and host, stores them as time series, draws dashboards and pages people when something is wrong. In this design metrics servers pull current values from each target on a fixed interval, using service discovery to know what exists, and store samples in a local time-series database. Two identical servers scrape the same targets so one can fail without gaps.
Alert rules run inside the metrics servers, and an alert manager de-duplicates, groups and routes the results to pagers and chat. For long retention, finished data blocks are shipped to object storage, compacted and downsampled, and a query layer merges recent data from the servers with older data from storage. This is a simplified learning design, not any company's monitoring stack.
Trade-offs and what changes at scale
Pull-based collection makes target health visible and keeps targets simple, but short-lived jobs and targets behind firewalls need a push path. Metrics are aggregates sampled every few seconds, so they suit alerting and trends, not exact per-request accounting such as billing. The main scaling limit is cardinality: each distinct label combination is a series held in memory, so high-cardinality labels are kept out. At larger scale targets are split across many server pairs by team or region, and the global query layer and downsampling keep long-range queries fast.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Application Services count requests and record latency histograms with a client library and expose the current values on a /metrics page. Each value is a series named by a metric and a small set of labels such as service, route and status.
The metrics server asks Service Discovery for the current list of instances instead of using a static file. Containers come and go all day, and a target missing from the list is a target nobody watches.
Metrics Server A pulls from every target, including the Node Exporters, every 15 seconds. Pulling means the server itself notices when a target stops answering, which a push model cannot tell apart from a quiet one.
Batch Jobs that finish in seconds push their final metrics to the Pushgateway, which is then scraped like any other target. This is the exception, kept for jobs that would end before a scrape reached them.
Scraped samples go to the Local TSDB: first a write-ahead log and an in-memory block, then compressed two-hour blocks on disk. A single server keeps a couple of weeks this way and needs no other database to run.
Metrics Server B scrapes the same targets independently, so losing one server loses no data and no alerts. The two copies differ slightly in timing and are de-duplicated at query time.
Recording & Alert Rules run on a schedule: recording rules store expensive aggregates as new series, and alert rules fire when a query such as error rate above 2% for 10 minutes stays true. The "for" duration filters out short spikes.
Both servers send firing alerts to every member of the Alertmanager Cluster, which de-duplicates them and groups related alerts into one notification. Sending to all members, rather than load balancing, keeps alerts flowing if one member is down.
Routing rules send critical alerts to the On-call Pager and warnings to Team Chat, and inhibition hides warnings while a related critical alert is firing. Silences let engineers mute alerts during planned maintenance.
An alert that always fires is routed to an external Dead Man's Switch, which pages if it stops arriving. That catches the case where the monitoring system itself is down and silent.
The Block Uploader copies each finished block to the Object Store, and the Compactor merges blocks and writes downsampled versions. Object storage is cheap enough to keep a year or more, which local disks are not.
The Global Query Layer sends recent ranges to both metrics servers and older ranges to the Store Gateway, merges the replicas and caches the result. Dashboards see one view across all servers and all retention.
Dashboards query only the global layer, so a panel spanning six months reads downsampled data instead of raw samples. Raw resolution is kept for recent data, where it is actually used.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (20 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Collection (HA pair)layer
- Local TSDB: WAL plus 2-hour blocks, ~15 days
- Metrics Server A: scrapes every target, evaluates rules
- Metrics Server B: identical replica of A
- Recording & Alert Rules: precomputed series, alert conditions
Monitored Targetslayer
- Application Services: expose /metrics via client library
- Batch Jobs: finish before a scrape would reach them
- Node Exporters: CPU, memory, disk per host
- Pushgateway: holds metrics pushed by batch jobs
Service Discovery
- Service Discovery: current list of scrape targets
Alertingdomain
- Alertmanager Cluster: dedup, group, route, silence
- Dead Man's Switch: alerts if heartbeats stop arriving
- On-call Pager: pages the engineer on call
- Team Chat: low-urgency notifications
Long-term Storagedomain
- Block Uploader: ships finished blocks
- Compactor: merges and downsamples old blocks
- Object Store: months to years of blocks (s3)
- Store Gateway: serves queries over stored blocks
Querydomain
- Dashboards: graphs and ad-hoc queries
- Global Query Layer: fans out, merges, de-duplicates
- Query Cache: split-by-day results (redis)
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Log aggregation pipelineHow a log pipeline collects lines from every host, buffers them in a topic, indexes only labels, and keeps compressed text in object storage.
- Time-series databaseHow a time-series database appends samples behind a write-ahead log, compresses them into immutable blocks, and answers range queries by label.
- Multi-tenant SaaS analytics pipelineHow a SaaS product collects events from many customers, keeps each tenant's data isolated, and serves fast dashboards from a columnar store.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.