SaaS & data, an interactive architecture walkthrough
Feature flag service: how the architecture works
How a feature flag service publishes rule snapshots that SDKs evaluate locally, rolls out by stable user buckets, and pushes kill switches in seconds.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A feature flag service lets teams turn features on or off, or roll them out to a percentage of users, without deploying code. Flag definitions and targeting rules are edited in a control plane with access control, approvals and an audit log, then published as a versioned ruleset snapshot for each environment.
Evaluation happens inside each service: a server SDK downloads the snapshot through a CDN, keeps it in memory and evaluates flags locally, while a streaming connection tells it when a new version exists. Percentage rollouts hash the flag key with the user's id, so each user consistently gets the same answer on every instance. Browsers and mobile apps call a relay proxy that returns evaluated values instead of the rules. Exposure events feed experiment analysis. This is a simplified learning design, not any vendor's architecture.
Trade-offs and what changes at scale
Local evaluation makes flag checks fast and keeps the product working when the flag service is down, at the cost of a short delay before changes reach every instance and the need to ship SDKs for each language. Server-side evaluation for clients protects rules but adds a network call. Every flag is a branch in the code, so the main long-term cost is flags nobody removes. At larger scale, snapshot size and the number of connected SDKs drive the design, so snapshots are split per environment or project and fanned out through the CDN.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
An engineer creates a flag in the Flag Dashboard with a default value and targeting rules, and the Flag Management API checks through SSO that they may edit production. Production changes can require a second person's approval, since a flag change is a production change.
Every change is written to the Audit Log with the author and the before and after values. During an incident, the first question is often which flag changed and when.
The Config Publisher turns the environment's flags into one versioned ruleset snapshot. Services download rules, not answers, so they can evaluate any user without asking the service.
The snapshot is written to the Snapshot Store and served through the Flag CDN. Thousands of service instances fetching it then put no load on the control plane.
At startup the Server SDK loads the snapshot, and the Backend Service then evaluates flags in-process in microseconds. If the flag service is unreachable, the SDK keeps its last snapshot or falls back to the coded default, so a flag outage never takes the product down.
When a new version is published, the Streaming Service tells connected SDKs to fetch it, and SDKs also poll as a fallback. A kill switch therefore reaches every instance within seconds rather than at the next deploy.
For a 10% rollout, the SDK hashes the flag key and user id into a bucket and enables the feature for buckets 0 to 9. The Backend Service gets the same answer for the same user on every instance without any shared state.
Web & Mobile Apps send their user context to the Relay Proxy and get back only the evaluated values. Shipping the full ruleset to a browser would expose targeting rules and customer segments.
Each time a user is shown a variant, the SDK records an exposure on the Exposure Events topic, de-duplicated per user and session. Experiments compare only users who were actually exposed.
Experiment Analytics joins exposures with product metrics such as conversion and compares the variants. Running the analysis outside the evaluation path keeps flag checks fast.
The Stale Flag Detector finds flags that have served one value to everyone for weeks and opens cleanup tickets through CI / CLI. Old flags left in code are a common source of untested paths.
Observability tracks the age of each SDK's snapshot and evaluation errors. An SDK stuck on an old version is evaluating rules nobody intended anymore.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Control Planedomain
- Audit Log: every change: who, what, when (postgresql)
- Config Publisher: builds a versioned ruleset snapshot
- Flag DB: flag definitions per environment (postgresql)
- Flag Management API: flags, rules, segments, approvals
- SSO: who may change which environment
Distributiondomain
- Flag CDN: serves snapshots worldwide
- Snapshot Store: ruleset JSON per environment (s3)
- Streaming Service: pushes "new version" notices (websocket)
Evaluationdomain
- Backend Service: checks flags on every request
- Relay Proxy: evaluates for untrusted clients
- Server SDK: in-process evaluation, local snapshot
- Web & Mobile Apps: receive evaluated values only
Experimentationdomain
- Experiment Analytics: metrics per variant (clickhouse)
- Exposure Events: flag, variant, user, time (kafka)
- Stale Flag Detector: finds flags fully rolled out
Observability
- Observability: snapshot age, evaluation errors
Flag Authorslayer
- CI / CLI: flags as code, stale-flag checks
- Flag Dashboard: create flags, rules, rollouts
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Multi-tenant SaaS analytics pipelineHow a SaaS product collects events from many customers, keeps each tenant's data isolated, and serves fast dashboards from a columnar store.
- Metrics and monitoring systemHow a monitoring system scrapes metrics from every service, alerts through grouped and routed notifications, and keeps years of data in object storage.
- Log aggregation pipelineHow a log pipeline collects lines from every host, buffers them in a topic, indexes only labels, and keeps compressed text in object storage.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.