Web fundamentals, an interactive architecture walkthrough
Distributed rate limiter: how the architecture works
How an API rate limiter shares token buckets across gateways, answers 429 with a retry hint, fails safely and sheds load when the system is stressed.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A rate limiter decides, for each request, whether the caller has used up its allowance. In a distributed system the decision has to be shared: a caller's requests arrive at many gateways, so the allowance lives in a fast shared store that every gateway updates atomically. Limits are defined per customer, route or plan, cached at the gateways, and answered with a clear 429 and a retry hint. A separate load shedder protects the service itself by looking at overall health rather than at any one caller.
Trade-offs and what changes at scale
Every check adds a network hop, so limiters either accept a little latency or count locally and sync periodically, trading exactness for speed. Cloudflare has published an edge design that counts locally and uses a sliding-window approximation, with a small, measured error rate. Atomic scripts keep one store correct, but hot keys concentrate load on one shard. Failing open favours availability, failing closed favours protection. Limits that are too tight break legitimate bursts, so new limits should be dark-launched and monitored. Rate limits stop abuse of a shared service but are not a substitute for authentication or for per-endpoint capacity planning. This is a simplified teaching model based on public information, not any company's actual architecture.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
A request arrives. The gateway works out who to charge it to (an API key or account, sometimes also an IP address) and which route it is for. The choice of key decides what the limit protects: one customer, one user, or the service as a whole.
The gateway asks the Limiter Client for a decision. It finds the rule for this key and route in a local cache of rules, so it needs no network call to learn the limit, and works out the request's cost (a heavy endpoint can cost more than one token).
Rules reach the Limiter Client from the Rules Service, which stores them in the Rules Store. An operator edits a limit or adds an override for one customer, and the change is pushed to every gateway without a deploy. A gateway that misses an update keeps using its older rules until the next one arrives.
The Limiter Client asks the Counter Store to consume tokens for the key. With a token bucket, the stored state is the token count and the time of the last refill, and the check, refill and decrement need to happen as one step. A server-side script (a Redis Lua script runs atomically) does this in a single round trip, so two gateways cannot both spend the last token.
Why a bucket and not a plain counter? A fixed-window counter lets a caller send a full allowance at the end of one window and again at the start of the next. A sliding window avoids that, and one cheap approximation blends the previous and current window counts. A token bucket allows short bursts up to its capacity while enforcing an average rate. Which one fits is a product decision about burst tolerance.
If the decision is 'allow', the gateway forwards the request to the Backend Service. Allowed responses can carry headers telling the caller how much allowance remains, so well-behaved clients slow down before they are rejected.
If the bucket is empty, the gateway answers 429 Too Many Requests, usually with a Retry-After header (RFC 6585), and a message that says which limit was hit. Clients should back off with randomised delays so rejected callers do not all retry at the same instant.
If the Counter Store is slow or down, the Limiter Client must choose. Failing open keeps the API available but removes protection for a while; failing closed protects the backend but turns a limiter outage into an API outage. Stripe's public post recommends that limiter errors, including a Redis outage, should not affect requests. A local in-memory allowance can cover short outages.
A separate check looks at the system, not the caller. The Load Shedder receives utilisation signals from the Backend Service and, when it is under strain, rejects lower-priority traffic first so critical requests still succeed. Shedding and recovery are made deliberately gradual so the system does not flap.
At scale the Counter Store is sharded by key, so one very busy key lives on one shard and becomes a hot spot. Options are splitting a key into several sub-buckets, or letting each gateway lease a batch of tokens and spend them locally, which cuts calls to the store at the price of an approximate limit.
Observability counts allowed, limited and failed-open decisions per rule. A new limit is best run in shadow mode first, logging what it would block without blocking, then tuned, and kept behind a kill switch so a wrong rule can be switched off quickly.
Sequence diagram
The same walkthrough as a UML sequence diagram: components left to right in the order a request reaches them, numbered by step. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Edgelayer
- API Gateway: identifies caller, enforces the decision
- Limiter Client: rule lookup, cost, calls the counter store
- Load Shedder: drops low-priority work when the fleet is stressed
Clientslayer
- Admin Console: operators edit limits and overrides
- API Client: apps and partners calling the API
Limits Planedomain
- Counter Store: token buckets or window counters, sharded by key (redis)
- Rules Service: limits per plan, tenant and route
- Rules Store: rule definitions and overrides
Protected APIlayer
- Backend Service: the API being protected
Observability
- Observability: allowed, limited and failed-open counts
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- API gatewayHow an API gateway authenticates, rate-limits and routes every request to healthy backends, with config pushed from a separate control plane.
- URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.
- Distributed cacheHow a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.