Web fundamentals, an interactive architecture walkthrough
API gateway: how the architecture works
How an API gateway authenticates, rate-limits and routes every request to healthy backends, with config pushed from a separate control plane.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
An API gateway is the single entry point between external clients and a set of backend services. Each request passes through an ordered chain of checks on a stateless gateway node: TLS termination and route matching, authentication, rate limiting, optional caching, and finally forwarding to a healthy backend instance with timeouts and retries. Clients see one stable API while services behind it can be split, moved or rewritten.
Configuration lives in a separate control plane. Operators change routes, keys and policies through an admin API, and the change is pushed to every gateway node, which keeps serving with its last known configuration if the control plane is unavailable. Access logs and traces leave the request path asynchronously. This is a simplified learning design, not any specific product's architecture.
Trade-offs and what changes at scale
Every request pays for the gateway hop, so filters must be cheap and avoid network calls; tokens are verified locally and global rate limits sit behind a local token bucket. Putting business logic or heavy response aggregation in the gateway makes it a shared bottleneck that every team must change, so that logic is usually kept in backend-for-frontend services instead. Global quotas depend on a shared counter store, and a decision is needed on whether to allow or reject traffic when it is unreachable. At larger scale gateways are deployed per region, and the same data plane is often reused for service-to-service traffic as a mesh.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Every external request first reaches the CDN / WAF, which absorbs volumetric attacks and drops requests matching known bad patterns. Filtering this early means the gateway's own capacity is spent on real traffic.
The L4 Load Balancer spreads connections across stateless gateway nodes. Because no request state lives on a node, nodes can be added or replaced without draining sessions.
On a gateway node, the Listener & Router terminates TLS and matches the host and path to a route, such as /v1/orders to the orders service. Routing in one place lets backend services move or split without clients changing URLs.
The Auth Filter verifies a JWT signature against keys fetched and cached from the Identity Provider, or looks up a partner's API key. Verifying tokens locally avoids a network call to the identity provider on every request.
The Rate Limit Filter applies a local token bucket per caller, which absorbs short bursts on each node. Requests over the limit get a 429 before any backend sees them.
For quotas that must hold across all nodes, such as a partner's requests per minute, the filter asks the Rate Limit Service, which counts in Rate Limit Counters. The local bucket in front keeps most traffic from needing this extra call.
Public GET responses that allow it are served from the Response Cache for a few seconds. Personalised responses are never cached here, so one user cannot see another's data.
The Upstream Pool picks a healthy instance from the list kept by the Service Registry, with a timeout, a retry budget and a circuit breaker per service. One slow backend then fails fast instead of tying up every gateway worker.
For a mobile order screen, the gateway can call the Orders Service and Catalog Service over gRPC in parallel and return one JSON response. This cuts the number of round trips a phone makes on a slow network.
Operators change routes and plugins through the Config API, which writes a new version to the Config Store, and the Config Distributor pushes it to every node. Nodes apply config without a restart and keep the last good version if the control plane is down.
Each request produces one access log event on the Access Log Stream, which loads into API Analytics for usage by key, route and status code. Logging asynchronously keeps it off the request's latency.
Gateway nodes send latency, error and retry metrics plus trace headers to Observability. The gateway starts the trace, so a slow request can be followed into whichever backend caused it.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (23 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Clientslayer
- Mobile App: iOS and Android
- Partner Integrations: third parties using API keys
- Web App: browser single-page app
Edge Networklayer
- CDN / WAF: DDoS absorption, bot and WAF rules
- L4 Load Balancer: spreads TCP across gateway nodes
Gateway Data Planeboundary
stateless gateway nodes, scaled out
- Auth Filter: JWT and API key checks
- Listener & Router: TLS, match host/path to a route
- Rate Limit Filter: local token bucket per key
- Response Cache: short-TTL cache for public GETs (redis)
- Upstream Pool: pick instance, timeouts, retries
Policy Servicesdomain
- Identity Provider: issues tokens, publishes signing keys
- Rate Limit Counters: per-key counters with TTL (redis)
- Rate Limit Service: global quotas across all nodes (grpc)
Control Planedomain
- Config API: routes, plugins, consumers, keys
- Config Distributor: pushes config to nodes (grpc)
- Config Store: versioned gateway configuration (postgresql)
- Service Registry: healthy endpoints per service
Backend Serviceslayer
- Catalog Service: products and prices (grpc)
- Orders Service: orders and payments (grpc)
- Users Service: accounts and profiles
Telemetrylayer
- Access Log Stream: one event per request (kafka)
- API Analytics: usage per key, route, status (clickhouse)
- Observability: metrics, traces, alerts
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed rate limiterHow an API rate limiter shares token buckets across gateways, answers 429 with a retry hint, fails safely and sheds load when the system is stressed.
- URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.
- Distributed cacheHow a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.