AI applications, an interactive architecture walkthrough
ML feature store: how the architecture works
How a feature store computes features once, keeps timestamped history for point-in-time training sets, and serves the latest values to live models.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
An ML feature store manages the inputs that models use, so that the same feature is computed once and served consistently to both training and live predictions. Features are defined as code in a registry with their entity, source and owner. Batch jobs compute slow-changing aggregates and streaming jobs compute rolling windows, writing full timestamped history to an offline store and the latest values to an online store.
Training sets are built with point-in-time joins: each labelled event gets the feature values as they were at that moment, which prevents future data from leaking into training. Live models read the latest values by key from the online store through a feature server in milliseconds. Monitoring checks freshness and drift. This is a simplified learning design, not any specific product's implementation.
Trade-offs and what changes at scale
Keeping two stores, one for history and one for low-latency lookups, duplicates data but serves two very different access patterns. Streaming features are fresh but costly to run and harder to backfill than batch features. A shared store reduces duplicated work across teams only if definitions are owned and documented, otherwise it fills with near-identical features. At larger scale the online store is sharded by entity key, and materialization is incremental because rewriting every entity each hour stops being affordable.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
A team defines features such as "card spend in the last hour" in the Feature Repo, and applying them records the entity, source, owner and schema in the Feature Registry. One definition is then used for both training and serving.
The Batch Pipeline computes slow-changing features, such as 90-day order counts, from the Warehouse and Application DB and writes them with timestamps to the Offline Store. Keeping every value with its time is what makes past states reproducible.
The Streaming Pipeline maintains rolling windows over the Event Stream and pushes the latest values straight into the Online Store within seconds. Fraud signals lose most of their value if they arrive an hour late.
The streaming job also appends each value to the Offline Store, so training sees the same streamed features the model will see live. Computing a feature two different ways for training and serving is the usual cause of skew.
The Materializer copies the latest batch values from the offline store into the Online Store on a schedule. The online store keeps only the current value per entity, which keeps lookups to a single key read.
A data scientist submits labelled events with timestamps from a notebook, and the Training Set Builder joins each one with feature values as of that moment. Without this as-of join, a model trained on today's values would look far better offline than it performs live.
The resulting training set goes to the Training Job, which records the feature names and versions used. Serving later requests exactly that list.
At checkout, the Model Service asks the Feature Server for a customer's and card's features, which it reads from the Online Store in a few milliseconds. Models get features by name and never query source databases during a request.
Access Control decides which teams may read which features, since some are derived from sensitive data. Permissions live with the definitions, so a new consumer must be granted access explicitly.
The Feature Monitor checks freshness, null rates and how live values compare with training distributions. A stale or drifting feature degrades predictions silently, so it is alerted on like an outage.
Observability tracks lookup latency and pipeline failures for every feature. Lookup latency adds directly to the latency of every prediction.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Definitionsdomain
- Access Control: who may read which features
- Feature Registry: entities, features, sources, owners
- Feature Repo: feature definitions as code in Git
Data Sourceslayer
- Application DB: orders, accounts (postgresql)
- Event Stream: clicks, payments, logins (kafka)
- Warehouse: historical aggregates (clickhouse)
Feature Computationdomain
- Batch Pipeline: daily and hourly aggregates
- Streaming Pipeline: rolling windows in seconds (flink)
Storesdomain
- Materializer: copies latest values offline -> online
- Offline Store: full feature history with timestamps (s3)
- Online Store: latest value per entity (redis)
Consumersdomain
- Data Scientist Notebook: explore and select features
- Feature Server: online lookup API (grpc)
- Model Service: fraud score at checkout
- Training Job: fits the model
- Training Set Builder: point-in-time joins
Monitoringdomain
- Feature Monitor: freshness, nulls, drift
- Observability: lookup latency, job failures
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Recommendation systemHow a recommendation system retrieves thousands of candidates from millions of items, ranks them with a model, and filters for freshness and diversity.
- Model inference servingHow a model serving platform batches requests on CPUs and GPUs, manages the LLM KV cache, scales GPU capacity from queue depth, and canaries versions.
- RAG chatbot over your docsHow a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.