The problem
Your company provides a public API used by hundreds of customers, each with their own usage limits. API traffic is served by a fleet of stateless servers behind a load balancer. To prevent abuse and ensure fairness, you must enforce per-customer rate limits, even as requests are routed to any server.
You are a backend engineer tasked with designing a distributed rate limiter that works seamlessly across all API servers.
If the design is wrong, customers may exceed their limits, risking infrastructure overload, or get unfairly blocked, hurting reliability and trust.
Constraint 1
No single point of failure in rate limiting.
Constraint 2
Rate limiting must not noticeably increase API latency.
Constraint 3
The solution must scale with traffic growth.
Constraint 4
All API servers must enforce limits consistently.
Functional Requirements
What the system must DO.
| Requirement | Priority |
|---|---|
| The system must enforce a configurable rate limit per customer. | Must have |
| Rate limits must be enforced consistently, regardless of which API server handles the request. | Must have |
| The time window for rate limiting must be configurable (e.g., per minute, per hour). | Nice to have |
| The rate limiting check must add minimal latency to each API request. | Must have |
| If the rate limiter backend is unavailable, the system should fail safely (e.g., reject or allow with warning). | Nice to have |
Non-Functional Requirements
How well it must do it — latency, availability, scale. Measurable, not vibes.
| Requirement | Priority |
|---|---|
| The system must scale to handle peak traffic and customer growth. | Must have |
| The rate limiting system must be highly available, with no single point of failure. | Must have |
| The system should provide metrics and logs for monitoring rate limit usage and errors. | Nice to have |
What you learn by asking
In an interview these facts are NOT volunteered — the candidate earns them with clarifying questions. These are the answers behind this design.
| If you ask about… | The answer |
|---|---|
| API request volume | Average 2,000 requests per second (QPS) across all customers, peaking at 10,000 QPS during busy hours. |
| Customer count | About 500 active customers today, expected to double in 2 years. |
| Rate limit granularity | Most customers have a limit of 1,000 requests per minute, but some have custom higher or lower limits. |
| API server fleet size | API is served by 20 stateless servers behind a load balancer. |
| Latency budget | Rate limiting logic must add less than 10 milliseconds to the 99th percentile API latency. |
| Data retention | Usage counters need only be stored for the current and previous rate limit window. |
Capacity Estimation
Rough arithmetic turns the requirements into engineering decisions — it tells you which parts need scaling machinery and which are trivially easy. Only the order of magnitude matters.
| Quantity | Answer | How it is derived |
|---|---|---|
| Total API requests per day (peak) | ~864M requests | 10,000 requests/sec × 60 sec/min × 60 min/hr × 24 hr/day = 864,000,000 requests/day |
| Rate limit checks per second (peak) | ~10K checks/sec | Each API request requires a rate limit check, so 10,000 requests/sec = 10,000 checks/sec |
| Storage needed for counters per window | ~4000 bytes | 500 customers × 8 bytes per counter = 4,000 bytes per window |
The architecture, one version at a time
Nobody designs the final diagram in one stroke. Strong candidates start with the simplest thing that works, then let the numbers break it: each version fixes exactly one bottleneck with exactly one new component.
At each break you get a chance to call the fix yourself before it is revealed. You can stop reading after any version and still hold a complete, working system in your head — that is the point.
v1 — The Simplest Thing That Works
The fix: Each API server tracks rate limits in its own memory.
The easiest approach is for each API server to maintain its own in-memory counters for each customer. This is fast and simple, adding almost no latency.
However, this design falls apart when requests for the same customer are routed to different servers: counters are not shared, so customers can easily exceed their limits by spreading requests across servers.
v1 — The Simplest Thing That Works — the picture
Where v1 breaks
At 10,000 QPS, local counters allow customers to exceed limits by routing requests to multiple servers.
Quick check
Which single component would you add to fix that?
v2 — each server counts on its own
The fix: Introduce a shared rate limiter service that all API servers query before serving requests.
To enforce limits consistently, add a distributed rate limiter service. Now, every API server checks and increments usage for each customer centrally.
This ensures correct enforcement, but the rate limiter can become a bottleneck or single point of failure if not designed carefully.
v2 — each server counts on its own — the picture
Where v2 breaks
Every one of the 10,000 requests/sec needs a check, and v2 routes all of them through a single rate limiter. That one process is both a ceiling on throughput and a single point of failure — the design says there must not be one.
Quick check
Which single component would you add to fix that?
v3 — one rate limiter is now the weak point
The fix: Run several rate limiter instances instead of one. That immediately raises a question: if each instance keeps its own counters, we are back to the v1 problem. So the counters move out into a shared counter store — a very fast in-memory store that every instance reads and updates, so they all enforce one number.
A check is now two cheap steps: the rate limiter asks the shared store for this customer's current count for the window and increments it. Because the store keeps everything in memory, that round trip costs well under a millisecond — which is what keeps the "no noticeable added latency" requirement satisfiable.
The increment has to be atomic. If two instances read 999 at the same moment, both would see room and both would allow — one request over the limit. A single read-and-increment operation executed by the store itself removes that window entirely.
The counters themselves are tiny: 500 customers × 8 bytes is about 4 KB per window, so a single modest store holds every customer comfortably. What is being scaled here is the checking, not the data.
The finished design
The full reference design, with the reason behind every connection. When you practice, you will rebuild this from a palette that includes decoys.
Why each connection exists
- API ClientLoad BalancerClients connect through the load balancer for traffic distribution.
- Load BalancerAPI ServerLoad balancer routes requests to any API server.
- API ServerDistributed Rate LimiterAPI server must check and update the customer's usage before serving the request.
- Distributed Rate LimiterShared Counter StoreThe counters cannot live inside the rate limiter instances — several of them run at once, so the count they enforce has to live in one shared store they all read and update.
The design in one glance
Each version fixed exactly one problem. If you can retell this table from memory, you can derive the whole architecture on a whiteboard.
| Version | What broke | The fix |
|---|---|---|
| v1 — The Simplest Thing That Works | — the starting point | Each API server tracks rate limits in its own memory. |
| v2 — each server counts on its own | At 10,000 QPS, local counters allow customers to exceed limits by routing requests to multiple servers. | Introduce a shared rate limiter service that all API servers query before serving requests. |
| v3 — one rate limiter is now the weak point | Every one of the 10,000 requests/sec needs a check, and v2 routes all of them through a single rate limiter. That one process is both a ceiling on throughput and a single point of failure — the design says there must not be one. | Run several rate limiter instances instead of one. That immediately raises a question: if each instance keeps its own counters, we are back to the v1 problem. So the counters move out into a shared counter store — a very fast in-memory store that every instance reads and updates, so they all enforce one number. |
API Design
Every operation traces back to a requirement — nothing extra, nothing missing.
| Endpoint | Purpose |
|---|---|
| POST /rate_limit/check | Checks if a customer is within their rate limit and increments their usage if allowed. |
| GET /rate_limit/status | Returns the current usage and limit for a customer. |
Data Model
| Entity | Fields |
|---|---|
| RateLimitCounter | customer_id: string (Unique identifier for the customer), window_start: timestamp (Start of the current rate limit window), count: integer (Number of requests made in the current window), limit: integer (Maximum allowed requests in the window) |
Defending the Design
Interviewers close by stress-testing the design. Strong answers follow one shape: which component → what breaks there → the fix and its cost. Here are this problem's probes and the points a strong answer hits:
What could make your distributed rate limiter a bottleneck, and how would you address it? Centralized state or locking can cause contention at high QPS.. Network latency between API servers and rate limiter can slow down requests.. Mitigate by sharding counters, using in-memory data stores, or optimizing atomic operations..
How should your API servers behave if the rate limiter becomes unavailable? Decide whether to fail open (allow requests) or fail closed (block requests).. Fail open risks abuse; fail closed risks false blocks.. Log and alert on failures for operator visibility.. Consider fallback to cached limits with clear warnings..
If customer count and QPS double, what changes (if any) are needed in your design? Ensure rate limiter backend can scale horizontally (add more nodes/partitions).. Monitor for hot spots if some customers have much higher traffic.. May need to shard counters or partition by customer ID..
What are the tradeoffs of using a local cache on each API server for rate limit counters? Improves latency by avoiding network hops.. Risks stale or inconsistent counters if multiple servers handle the same customer.. Could allow over-limit usage if cache is not synchronized.. Best for read-heavy, low-write, or low-concurrency scenarios..
Tip — Key takeaway
Distributed rate limiting requires careful coordination to avoid both over-counting and under-counting, especially as API servers scale horizontally. The core challenge is enforcing per-customer limits with low latency and high consistency, without introducing a single point of failure. Sharding, in-memory datastores, and monitoring are key to balancing speed, correctness, and reliability.
Common trap — Common mistakes on this problem
Relying on a local in-memory cache per server, which causes inconsistent enforcement when requests are load balanced. · Using a single database instance for counters, creating a bottleneck and single point of failure. · Not handling rate limiter backend failures gracefully, leading to outages or silent abuse.
You've seen the whole design — now build it yourself
The practice run walks the same five steps, but YOU do the work: gather the requirements, run the numbers, wire the architecture — with an AI interviewer and per-step feedback.