SaaS & data, an interactive architecture walkthrough
Log aggregation pipeline: how the architecture works
How a log pipeline collects lines from every host, buffers them in a topic, indexes only labels, and keeps compressed text in object storage.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A log aggregation pipeline collects log lines from every container and host, stores them centrally and lets engineers search them during incidents. Agents on each node tail log files, add labels and buffer locally; a collector gateway redacts sensitive fields and routes by tenant; and a durable topic absorbs bursts before the storage tier.
The storage tier in this design indexes only a few labels per stream and keeps the log text in compressed chunks in object storage. Queries choose streams by label and then scan the text, with a query frontend splitting long time ranges across many queriers. A small subset, audit logs, is also sent to a full-text index, and all raw logs are archived for compliance. This is a simplified learning design, not any company's logging platform.
Trade-offs and what changes at scale
Indexing only labels keeps ingestion and storage cheap but makes searches over large time ranges slower than a full-text index, so the two are combined for the logs that need fast search. Putting a topic in front of storage adds a hop and some delay but stops bursts from turning into dropped lines. High-cardinality labels such as request ids create too many streams and are kept in the log text instead. At larger scale the cost is driven by volume, so sampling, per-tenant limits and retention policies matter more than adding query capacity.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Application Containers write structured lines to stdout, and a Node Agent on each host tails those files, parses them and attaches labels such as service and namespace. Collecting on the node means applications need no logging client of their own.
The Node Agent sends batches to the Collector Gateway, buffering on local disk if the gateway is slow. The gateway removes secrets and personal data, samples noisy debug logs and tags each record with its tenant.
Records go to the Log Topic before indexing. A deploy that suddenly logs ten times more, or a slow storage backend, then fills the topic instead of making agents drop lines.
The Distributor checks each stream against per-tenant rate and size limits and hashes its label set to pick the ingesters that own it. Hashing whole streams keeps one stream's lines together and in order.
Ingesters append lines to an in-memory chunk per stream and keep copies on more than one ingester, acknowledging after a quorum. Compressed chunks of a single stream compress well because neighbouring lines look alike.
Full or idle chunks are flushed to the Chunk & Index Store in object storage, together with an index from label sets to chunks. Only labels are indexed, so storage costs stay close to the size of the compressed text.
A search from the Log Explorer reaches the Query Frontend, which splits a 24-hour query into smaller time ranges and reuses any cached in the Results Cache. Splitting lets many queriers work on one search in parallel.
Queriers fetch the latest lines from the ingesters and older chunks from the store, then filter the text for the search terms. Searching text without a full-text index is slower per byte but avoids indexing every word ever logged.
The Compactor merges small index files and deletes chunks older than each tenant's retention period. Retention is the main cost control, so it is set per tenant and per stream type.
Audit logs, a small fraction of the volume, are also consumed by the Audit Indexer into a full-text Audit Search Index. Security investigations need fast arbitrary-term search on that subset, which would be too costly for all logs.
A copy of every raw record lands in the Archive Bucket on a cheap storage tier for compliance. It can be replayed into the pipeline later if a past period has to be searched again.
Log Alert Rules run queries such as count of "payment failed" lines per minute and page the On-call Pager above a threshold. This covers failures that show up in logs but have no metric yet.
Observability tracks lines received, lines dropped by agents and query latency. Dropped lines are the number that matters most, because a missing log is only noticed when someone needs it.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (19 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Collectionlayer
- Collector Gateway: redacts, samples, routes by tenant
- Log Topic: absorbs bursts, replayable (kafka)
- Node Agent: tails files, parses, adds labels, buffers
Log Sourceslayer
- Application Containers: write structured logs to stdout
- Hosts & Proxies: syslog, access logs
Write Pathdomain
- Chunk & Index Store: compressed chunks plus label index (s3)
- Compactor: merges index files, applies retention
- Distributor: validates, rate-limits, hashes streams
- Ingesters: build compressed chunks per stream
Read Pathdomain
- Log Explorer: search and live tail
- Queriers: read ingesters and the store, filter lines
- Query Frontend: splits queries by time, queues them
- Results Cache: cached query results (redis)
Other Consumersdomain
- Archive Bucket: raw logs kept for compliance (s3)
- Audit Indexer: full-text index of audit logs only
- Audit Search Index: security and compliance search (elasticsearch)
- Log Alert Rules: counts error patterns, fires alerts
- On-call Pager: receives log-based alerts
Observability
- Observability: ingest rate, dropped lines, query latency
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
- Grafana Loki docs: Architecture (write path, read path, object storage)
- grafana/loki: indexing labels instead of log contents
- fluent/fluent-bit: telemetry agent for collecting and forwarding logs
- OpenTelemetry Collector: vendor-agnostic receive, process and export
- Apache Kafka docs: Design (durable log, retention, consumer replay)
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Metrics and monitoring systemHow a monitoring system scrapes metrics from every service, alerts through grouped and routed notifications, and keeps years of data in object storage.
- Multi-tenant SaaS analytics pipelineHow a SaaS product collects events from many customers, keeps each tenant's data isolated, and serves fast dashboards from a columnar store.
- Time-series databaseHow a time-series database appends samples behind a write-ahead log, compresses them into immutable blocks, and answers range queries by label.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.