Rate limiter is a favorite system design question. Learn how to clarify the problem, compare token bucket and window counters, and handle many servers.
A rate limiter controls how many requests a user can make in a given time. It protects a service from abuse, accidental overload and runaway scripts. The question is popular because the basic idea is easy, which leaves plenty of room to show how you think.
If you have not seen our URL shortener walkthrough, read it too. The structure of the answer is the same.
Step 1: Clarify before you design
Spend the first two minutes asking questions. It shows maturity and it narrows a huge problem.
- What are we limiting: requests per user, per IP address, or per API key?
- What is the rule, for example 100 requests per minute?
- Is this for one server or a fleet of servers?
- What should happen when the limit is hit? Reject, queue, or slow down?
- How strict must it be? Is a small amount of error acceptable?
Step 2: Where does it live?
The limiter sits in front of your service, checking each request before it reaches your business logic. Common places are an API gateway or a small piece of middleware. It keeps counters in a fast shared store such as Redis.
When a request is blocked, return HTTP 429 Too Many Requests. Add headers such as Retry-After so well behaved clients know when to come back.
Step 3: Pick an algorithm
You should be able to explain at least two and compare them. These are the usual choices:
| Algorithm | How it works | Good | Watch out |
|---|---|---|---|
| Fixed window | Count requests in each calendar minute, reset at the boundary | Very simple, tiny memory | A burst across the boundary can double the limit |
| Sliding window log | Store a timestamp for every request, count those in the last minute | Exact | Memory grows with traffic |
| Sliding window counter | Blend the current and previous window counts | Good accuracy, small memory | An approximation |
| Token bucket | A bucket refills at a steady rate, each request spends a token | Allows short bursts, easy to tune | Two numbers to choose |
Token bucket is a strong default answer. Picture a bucket holding up to 10 tokens that gains 1 token per second. Each request takes one token. A quiet client builds up tokens and can burst. A busy client is held to one request per second on average.
Step 4: Sketch the logic
import time
class TokenBucket:
def __init__(self, capacity, refill_per_sec):
self.capacity = capacity
self.refill = refill_per_sec
self.tokens = capacity
self.last = time.time()
def allow(self):
now = time.time()
# add tokens for the time that has passed
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return FalseIn an interview you rarely need to write this out. Showing you understand the refill calculation is enough.
Step 5: The hard part is many servers
With one server the counter lives in memory and everything is easy. With ten servers, each would count separately, and a user could send ten times the limit. The counters must be shared, which is why Redis is common here.
That raises a new problem. Two servers may read the same counter at the same moment, both see room, and both allow a request. This is a race condition. The usual answer is to make “read, check, update” a single atomic step, for example with a small Lua script in Redis, or with atomic increment commands.
Step 6: Finish strong
- Summarize the design in two sentences.
- Mention monitoring: how many requests are being blocked, and for whom?
- Mention different limits per plan, such as free and paid users.
- Mention that limits should be visible to clients through response headers.