AI applications, an interactive architecture walkthrough
LLM gateway and router: how the architecture works
How an LLM gateway gives every team one API, enforces token limits and budgets, routes to the right model, and falls back when a provider fails.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
An LLM gateway sits between a company's applications and the language model providers it uses. Every service calls one OpenAI-style API with a virtual key; the gateway authenticates the key, applies token-based rate limits and content policies, optionally answers from a semantic cache, picks a model and provider, and translates the request into that provider's API using credentials the callers never see.
Routing uses aliases such as "fast" or "best", so models can be changed in configuration, and failed or throttled calls fall back to a second provider. Each call records tokens, latency and cost, which feed per-team spend reports and budget enforcement. This is a simplified learning design, not any specific product's implementation.
Trade-offs and what changes at scale
The gateway adds a hop and becomes critical infrastructure, so it must be stateless, replicated and fast, with limits checked against a shared counter store. Fallbacks improve availability but different models give different answers, so some features pin a model instead. Semantic caching saves cost on repeated questions but can return a wrong answer for a similar-looking prompt, so it is opt-in. At larger scale the gateway also routes to self-hosted models and tracks provider rate limits centrally, since many teams share the same provider quota.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Product Services call the gateway with one OpenAI-style API and a virtual key, and Key Auth maps that key to a team, a budget and allowed models. Teams never hold real provider keys, so a leaked virtual key can be revoked in one place.
The Token Rate Limiter charges the request's estimated tokens against the key's bucket in Limit Counters and corrects it after the response. Limiting by tokens, not requests, matches how providers bill and throttle.
The Policy Guard redacts personal data and blocks content that policy forbids before anything leaves the company. Doing this once in the gateway is more reliable than in every service.
For endpoints that allow it, the Semantic Cache returns a stored answer when a new prompt's embedding is very close to an earlier one. A similarity threshold that is too loose returns answers to different questions, so caching is opt-in per use case.
The Router resolves a model alias such as "fast" or "best" to a concrete model and provider, preferring the cheapest one that meets the task's tier. Callers name capabilities, so models can be swapped without code changes.
Provider Adapters translate the request into the chosen provider's API, attach the provider key from Provider Credentials and stream the response back. Differences between providers stay inside the adapters.
If Hosted Provider A returns rate-limit or server errors or times out, the router retries on Hosted Provider B with an equivalent model. A second source turns a provider outage into slightly different answers instead of failures.
Large batch workloads and sensitive data can be routed to Self-hosted Models on the company's own GPUs. That trades model quality for cost control and data that never leaves the network.
Every call emits a usage record with tokens, latency and cost to Usage Events, loaded into the Spend Store. Spend is then visible per team, model and product feature.
The Budget Enforcer compares spend with each key's budget and disables or downgrades keys that exceed it. Spend lags by seconds, so budgets are a soft cap backed by rate limits.
Platform engineers change routes, aliases and budgets through the Config API, and gateway nodes reload them without a deploy. Moving traffic to a new model is a config change that can be rolled back.
Observability tracks latency, error rates and fallback counts per provider and model. A rising fallback rate is the first sign a provider is degrading.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (19 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Control Planedomain
- Config API: routes, keys, budgets, policies
- Config DB: keys, teams, model aliases (postgresql)
- Provider Credentials: provider API keys, never given to callers
Gateway Data Planeboundary
one OpenAI-style API for every model
- Key Auth: virtual keys mapped to team and budget
- Limit Counters: per-key token buckets (redis)
- Policy Guard: PII redaction, blocked content
- Provider Adapters: one API in, provider APIs out
- Router: pick model, retries, fallbacks
- Semantic Cache: answers for near-identical prompts (redis)
- Token Rate Limiter: requests and tokens per minute
Model Providerslayer
- Hosted Provider A: frontier models
- Hosted Provider B: second source for fallback
- Self-hosted Models: open-weight models on own GPUs (grpc)
Callerslayer
- Batch Jobs: large offline workloads
- Internal Tools: notebooks, prototypes
- Product Services: chat, summarise, extract
Usage & Spenddomain
- Budget Enforcer: disables keys over budget
- Observability: latency, error rate per provider
- Spend Store: cost by team, model, feature (clickhouse)
- Usage Events: tokens, latency, cost per call (kafka)
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- AI agent platform with toolsHow an AI agent platform runs a model-and-tools loop on workers, checkpoints each step, scopes tools to the user, and pauses risky actions for approval.
- AI code review botHow an AI code review bot queues pull request webhooks, gathers diff context in a sandbox, verifies model findings, and posts inline comments.
- RAG chatbot over your docsHow a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.