AI applications, an interactive architecture walkthrough
Model inference serving: how the architecture works
How a model serving platform batches requests on CPUs and GPUs, manages the LLM KV cache, scales GPU capacity from queue depth, and canaries versions.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
Model inference serving runs trained models behind an API with predictable latency and cost. Requests pass through a gateway with quotas, a router that splits traffic between model versions, CPU-side pre- and post-processing, and a per-model queue with deadlines, before reaching model servers that batch requests to use their hardware well.
Small models are served on CPUs with dynamic batching. Large language models run on GPU servers that use continuous batching, so new requests join between token steps, and manage the attention KV cache in pages so more requests fit in memory. An autoscaler adds GPU nodes from queue depth, weights are cached on each node, and new versions roll out through shadow traffic and metric-checked canaries. This is a simplified learning design, not any specific platform's implementation.
Trade-offs and what changes at scale
Larger batches raise throughput and lower cost per request but add latency, so interactive and batch traffic are served from separate pools. GPU memory for the KV cache caps concurrency, which is why quantisation and prefix sharing matter. Keeping warm capacity avoids slow cold starts but pays for idle GPUs. At larger scale models are split across several GPUs, routing considers which server already holds a request's prompt prefix, and capacity is planned per model and region.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Product Services send requests to the Inference Gateway, which checks quotas and returns cached results for deterministic requests. Batch Scoring uses a separate, lower-priority quota so offline jobs cannot crowd out live traffic.
The Model Router sends most traffic to the stable version and a small share to a canary. Callers name a model, not a server, so versions can change underneath them.
Pre/Post Processing validates inputs, tokenises text and later formats outputs, on CPUs. Keeping this off the GPU servers leaves expensive accelerators doing only model work.
Requests wait briefly in the model's Request Queue with a deadline, and those that cannot be served in time are rejected early. Failing fast is better than answering after the caller has given up.
CPU Model Servers group requests that arrive within a few milliseconds into one batch for small models such as classifiers. Batching raises throughput a lot for a small added delay.
GPU LLM Servers use continuous batching: new requests join the running batch between token steps instead of waiting for the whole batch to finish. GPUs stay busy even when responses have very different lengths.
Each running sequence keeps its attention keys and values in the KV Cache, managed in small pages so memory is not wasted, and shared prompt prefixes are reused. GPU memory for this cache, not compute, usually limits how many requests run at once.
The Autoscaler adds GPU nodes when queues grow and removes them when idle, using queue depth rather than CPU load. GPU machines take minutes to start, so some headroom is kept warm.
A new node loads weights from its Local Model Cache, pulling them from the Artifact Store only on first use. Weights can be tens of gigabytes, so loading time dominates how fast capacity can be added.
The Deployer rolls out an approved version from the Model Registry by raising its traffic share step by step while checking latency and quality metrics. If the canary is worse, the share returns to zero automatically.
Before a canary, a Shadow Deployment can receive a copy of live traffic whose answers are logged but never returned. That tests a new model on real inputs without affecting users.
Sampled inputs and outputs flow through Inference Logs into the Monitoring Store, where quality and input drift are tracked per version, while Observability tracks latency and GPU use. A model can be healthy as a service and still be getting worse answers.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (19 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Inference Front Doordomain
- Inference Gateway: auth, quotas, model routing
- Model Router: version split: stable, canary, shadow
- Pre/Post Processing: tokenise, validate, format outputs
- Response Cache: deterministic requests (redis)
Model Runtimeboundary
one pool per model and hardware type
- CPU Model Servers: small models, dynamic batching (grpc)
- GPU LLM Servers: continuous batching, streaming tokens
- KV Cache: attention keys/values in GPU memory
- Request Queue: per-model queue with deadlines (redis)
- Shadow Deployment: new version, responses discarded
Capacitydomain
- Autoscaler: scales on queue depth and GPU load
- GPU Node Pool: accelerator machines
- Local Model Cache: weights on node disk
Model Lifecycledomain
- Artifact Store: model weights (s3)
- Deployer: canary rollout with metric checks
- Model Registry: versions, metrics, approvals (postgresql)
Telemetrydomain
- Inference Logs: sampled inputs, outputs, latency (kafka)
- Monitoring Store: quality and drift per version (clickhouse)
- Observability: latency, tokens/sec, GPU use
Callerslayer
- Batch Scoring: large offline jobs, cheaper tier
- Product Services: real-time predictions and generation
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
- vllm-project/vllm: LLM serving (PagedAttention, continuous batching, prefix caching)
- triton-inference-server/server: multi-framework inference server with dynamic batching
- kserve/kserve: model serving on Kubernetes (autoscaling, canary rollouts)
- argoproj/argo-rollouts: canary releases with metric analysis and automatic rollback
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- ML feature storeHow a feature store computes features once, keeps timestamped history for point-in-time training sets, and serves the latest values to live models.
- RAG chatbot over your docsHow a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
- AI agent platform with toolsHow an AI agent platform runs a model-and-tools loop on workers, checkpoints each step, scopes tools to the user, and pauses risky actions for approval.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.