Design a Rate Limiter

In my previous article, System Design: The Architecture That Scales , we looked at what happens when a simple application starts serving millions of users.

We added load balancers. We introduced multiple application servers. We moved data into databases and caches. We distributed the workload across machines.

But there is a problem hiding underneath all of that scaling.

What happens when the traffic itself becomes the problem?

Imagine an API that normally handles 10,000 requests per second. Everything is healthy. Latency is predictable. The database is comfortable.

Then, suddenly, one client starts sending 100,000 requests per second.

Maybe it is an attacker. Maybe it is a badly written script. Maybe the client simply experienced a traffic spike.

The reason almost doesn't matter.

Your infrastructure still has to deal with every request.

CPU starts climbing. Database connections are consumed. Queues begin growing. Latency increases.

And eventually, the traffic generated by one client starts affecting everyone else.

We could keep adding servers.

But at some point, the better question is not 'How do we process more traffic?'

It is:

That is the problem a rate limiter solves.

What Are We Actually Trying to Control?

Before choosing an algorithm or a database, let's make the problem precise.

A rate limiter controls how much traffic a client is allowed to generate within some constraint.

For example:

Or:

Or even:

Once the client exceeds that limit, we need to make a decision.

That's the entire job of the component.

At first glance, it looks almost trivial.

If we have one server, it actually is.

We could simply keep a counter in memory and increment it whenever a request arrives.

But the moment our application becomes distributed, that simple counter stops being enough.

Why Do We Need a Rate Limiter?

The obvious answer is abuse prevention.

But that is only one part of the problem.

Think about a shared system serving thousands or millions of clients.

If one client consumes a disproportionate amount of capacity, everyone else can suffer.

So the goal isn't simply to reject users.

The goal is to protect the system so that legitimate users can continue to use it.

Rate Limiting Isn't Load Shedding

At this point, it is tempting to think:

That's reasonable, but it introduces an important distinction.

Rate limiting and load shedding solve different problems.

Rate limiting asks:

Load shedding asks:

Imagine the database suddenly becomes slow.

A client might be perfectly within its configured rate limit, but the system itself could still be running out of capacity.

In that situation, load shedding can protect critical traffic by rejecting lower-priority work.

So we'll keep the two concepts separate throughout this design: the rate limiter controls who can send how much , while load shedding reacts to the current health of the system.

Before We Design It, Define the Contract

Now that the problem is clear, let's decide what a production rate limiter should actually guarantee.

• Accurate: enforce the configured limit without unnecessarily rejecting legitimate traffic.
• Low latency: rate-limit checks happen on the request path, so they must be cheap.
• Memory efficient: state may need to be maintained for millions of clients.
• Distributed: multiple servers must enforce one logical limit.
• Fault tolerant: a rate-limiter failure should not accidentally take down the API.
• Predictable client behavior: clients should know when they are throttled and how to respond.

These requirements give us something important.

A way to judge every design decision that comes next.

Where Should the Rate Limiter Live?

We know what the component needs to do.

The next question is where to put it.

Put It in the Request Path

Instead of implementing rate limiting separately inside every API handler, we can move it into middleware or an API gateway.

Now every request passes through the same logical enforcement point before reaching the application.

The application itself doesn't need to know how requests are counted.

But we still haven't solved the shared-state problem.

We need somewhere to store the state that all of these servers can access quickly.

For a typical application, that might mean a shared datastore such as Redis.

But before choosing Redis, there is another question we need to answer:

How should we count requests?

How Should We Count Requests?

This is where rate-limiting algorithms enter the picture.

There isn't one universally correct algorithm.

Each algorithm makes a different trade-off between accuracy, memory, burst handling, and complexity.

• Fixed Window Counter
• Sliding Window Log
• Sliding Window Counter
• Token Bucket
• Leaky Bucket

Rather than comparing them abstractly, let's build them one at a time.

Start Simple: Fixed Window

Suppose we allow 100 requests per minute.

The simplest solution is to divide time into fixed intervals and maintain a counter for each interval.

At the start of the next minute, the counter resets.

It's simple. It's cheap. And for many applications, that may actually be enough.

But simplicity hides a problem.

What if a client sends 100 requests at 12:00:59 and another 100 at 12:01:00?

The limiter sees two different windows and allows all 200 requests.

We just created a burst at the exact boundary between two windows.

So we need a more accurate view of time.

Remove the Boundary: Sliding Window Log

Instead of grouping requests into fixed buckets, what if we simply remembered when every request happened?

When a request arrives, we look at the timestamps from the previous window and count how many are still inside it.

This solves the boundary problem.

But we traded one problem for another.

Every request now creates state.

With millions of clients and high request rates, storing all of those timestamps can become expensive.

We need something that keeps the sliding-window behavior without remembering every request.

Approximate It: Sliding Window Counter

Instead of storing every timestamp, we can keep only aggregate counters.

Suppose the limit is 50 requests per minute.

The previous minute saw 42 requests. The current minute has seen 18, and the current minute started 15 seconds ago.

We're estimating how many requests from the previous window still belong to the current sliding window.

It isn't perfectly accurate.

But that's the point.

We deliberately accept a small approximation in exchange for dramatically lower memory usage.

This gives us an important system-design lesson:

What If Bursts Are Legitimate?

There is another problem our previous algorithms don't express particularly well.

What if short bursts are actually legitimate?

Imagine an API where the normal rate is 10 requests per second, but occasionally a client legitimately needs to send 50 requests at once.

We don't necessarily want to reject that burst.

This is where Token Bucket becomes useful.

Instead of counting requests directly, we maintain a bucket of tokens.

Tokens are added at a fixed rate. Each request consumes one.

The bucket also has a maximum capacity.

That capacity is what gives us controlled bursts.

For example, a bucket might hold 100 tokens and refill at 10 tokens per second.

A client can immediately consume the accumulated 100 tokens, but once they are gone, the long-term rate is controlled by the refill speed.

This makes Token Bucket particularly useful when we care about both average traffic rate and short-lived bursts.

What If We Want to Smooth the Traffic?

Token Bucket allows controlled bursts.

But sometimes we want the opposite behavior.

We want requests to leave the system at a predictable rate, regardless of how quickly they arrived.

That's the idea behind Leaky Bucket.

Requests enter the bucket and leave at a controlled rate.

If requests arrive faster than they can leave, they accumulate. Once the bucket is full, additional requests are rejected or otherwise handled according to the policy.

So now we have several algorithms, each answering a slightly different question.

For a normal application, this may already be enough.

But then we run into the problem that makes large-scale rate limiting really interesting.

What Changes at Internet Scale?

Imagine our application has grown from three servers to hundreds or thousands of machines spread across the world.

Suddenly, the question is no longer just:

The bigger question becomes:

A centralized counter sounds attractive.

Every server could simply ask the same datastore whether the request should be allowed.

But now every request depends on that datastore.

Network latency enters the critical path. The datastore becomes a bottleneck. And if it fails, the rate limiter may fail with it.

At large enough scale, the rate limiter itself can become the problem it was designed to prevent.

This is where looking at a system like Cloudflare becomes useful.

Move the Decision Closer to the Traffic

Cloudflare operates at the edge of the network, in front of the origin infrastructure.

That gives them an important advantage.

If excessive traffic can be identified at the edge, it can be stopped before reaching the origin.

This means the origin doesn't have to spend CPU, memory, database connections, or network bandwidth processing requests that are eventually going to be rejected anyway.

But now another distributed-systems problem appears.

Cloudflare has many edge locations.

How can they coordinate rate-limit state without maintaining one giant global counter?

Use the Network to Reduce the Coordination Problem

Cloudflare uses Anycast routing, where the same IP address can be announced from multiple locations.

Traffic is generally routed toward an appropriate nearby Point of Presence, or PoP.

That gives us a useful property.

Traffic from a particular client tends to remain within the same PoP under normal routing conditions.

So instead of trying to maintain one global counter, the counting problem can be localized.

We have reduced the scope of coordination.

Instead of:

we can think in terms of:

That is a much easier distributed problem to solve.

But a PoP Still Has Many Servers

We have reduced the problem, but we haven't eliminated it.

A single PoP can still contain multiple servers.

If every server keeps counters only in its own memory, requests from the same client can still produce inconsistent results.

We therefore need shared state inside the PoP.

One approach is to shard the state across machines so that a particular key is consistently associated with a particular shard.

Consistent hashing can help keep keys associated with the same shard even as machines are added or removed.

The important idea isn't the specific hashing algorithm.

It is this:

The Final Optimization: Get Expensive Work Off the Hot Path

We now have a distributed design.

But there is still one uncomfortable question.

What happens when the traffic itself becomes enormous?

If every request has to update a shared counter synchronously, the counter infrastructure can become the bottleneck.

At extreme traffic volumes, even a very fast datastore can be overwhelmed by the number of rate-limit operations.

This leads to an important optimization:

Instead of synchronously performing expensive counting work for every request, mitigation state can be propagated asynchronously.

Once the system determines that a client should be throttled, servers inside the PoP can receive a small piece of state telling them that mitigation is active and when it expires.

The request path then becomes extremely cheap:

The expensive coordination happens elsewhere.

This is one of the most important lessons from the entire design.

What Would We Actually Build?

Most applications do not need an architecture designed for internet-scale edge traffic.

For a typical API, we can start much simpler.

The middleware identifies the client, reads the relevant rate-limit state, applies the chosen algorithm, and makes the allow-or-reject decision.

Redis is a natural fit because rate-limit state is generally short-lived and needs very fast reads and updates.

This gives us a practical starting point without introducing the complexity of a globally distributed edge system.

What Happens When the Rate Limiter Breaks?

Our design protects the application from excessive traffic.

But we have introduced another dependency into the request path.

So here's the uncomfortable question:

What happens if Redis goes down?

If every request needs the rate limiter and the rate limiter needs Redis, then a Redis failure can accidentally become an API outage.

That would be ironic.

The component we introduced to improve availability could become the reason the service is unavailable.

One possible strategy is to fail open : temporarily allow traffic when the rate limiter cannot make a decision.

For security-sensitive endpoints, we may instead prefer to fail closed.

Neither choice is universally correct.

The important thing is that the behavior is a deliberate design decision rather than an accidental consequence of an exception.

What Should the Client Experience?

Eventually, some requests will be rejected.

The client needs to know why.

The response can tell the client that it has exceeded the allowed request rate and should retry later.

This matters particularly for automated clients.

A good rate limiter doesn't simply say "no."

It gives the client enough information to behave correctly.

A Correct Rate Limiter Can Still Break Your Users

There is one final production problem that isn't visible in the algorithm.

Imagine we configure:

The number looks reasonable.

We deploy it.

And suddenly one of our largest customers starts receiving 429s.

The rate limiter is working exactly as designed.

But the system is still broken from the customer's perspective.

That's why rate limiting should be introduced gradually.

Feature flags, metrics, alerts, and a way to disable the limiter are just as important as the algorithm itself.

One particularly useful technique is dark mode.

We calculate which requests would have been rejected without actually rejecting them.

Now we can observe real traffic and tune the limits before they start affecting users.

Putting It All Together

Let's go back to where we started.

One client was sending too much traffic.

We wanted to protect the system without hurting legitimate users.

That simple requirement led us through a surprisingly large number of design decisions.

We had to decide:

• where the limiter should live
• how requests should be counted
• where the state should live
• how multiple servers should coordinate
• how bursts should be handled
• what happens when dependencies fail
• how clients should respond
• how the limiter should be rolled out safely

For a typical application, the architecture might look like:

The rate limiter maintains short-lived state in a shared store such as Redis, while the algorithm determines how that state is interpreted.

If the requirements are simple, a fixed window may be enough.

If we need better temporal accuracy, sliding-window approaches become attractive.

If controlled bursts matter, Token Bucket becomes a strong candidate.

And if we're operating at internet scale, the problem changes again.

At that point, choosing an algorithm is only one part of the problem. We also need to think about placement, sharding, locality, asynchronous coordination, and the cost of the hot path.

Conclusion

Rate limiting looks deceptively simple.

At first, it seems like all we need is a counter and anif statement.

Then the system grows.

The counter becomes distributed state. Distributed state creates coordination problems. Coordination introduces latency. Scale makes the datastore itself a bottleneck. And failures force us to decide whether protecting the system is worth temporarily weakening the limit.

That's what makes rate limiting such a useful system-design problem.

The difficult part isn't writing the counter.

The difficult part is deciding where that counter lives, how accurate it needs to be, how much coordination we can afford, and what happens when everything around it starts failing.

And eventually, the goal becomes clear:

That's the real lesson behind designing a rate limiter.

Start with a counter.

Then follow the problems that counter creates.

That's where system design gets interesting.

Start simple. Follow the problems. Design for scale.