Designing a Rate Limiter: From One Server to a Fleet

"Design a rate limiter" shows up in almost every system design interview, and for good reason. It's small enough to finish in an hour but touches a lot of real problems: choosing an algorithm, shared state, clock skew, failure modes and API design.

It's also something you'll actually build. Any public API, login endpoint or expensive background job eventually needs one. This post goes through how I think about it, starting with a single process and ending with a fleet of servers sharing limits through Redis.

First, what are we protecting?

Before choosing an algorithm, answer three questions:

1. Who is being limited? A user ID, an API key, an IP address, or a combination. IP-based limits punish everyone behind the same corporate NAT, so prefer authenticated identities where you have them.
2. What is the limit protecting? Your database's capacity, an upstream quota, fairness between tenants, or abuse prevention. The answer tells you how strict and how precise the limit has to be.
3. What happens when the limiter itself is down? More on this at the end. It's the question most designs forget.

The algorithms

Fixed window counter

Count requests per key in fixed time buckets, such as "requests between 12:00:00 and 12:00:59", and reject once the count passes the limit.

It's easy to implement and cheap: one counter per key per window. The weakness is at the edges. With a limit of 100 per minute, a client can send 100 requests at 12:00:59 and another 100 at 12:01:00, which is 200 requests in about two seconds. For abuse prevention that's often fine. For protecting a fragile dependency it isn't.

Sliding window log

Store a timestamp for every request and count the ones within the last N seconds. This is exact, but memory grows with traffic: 10,000 requests per minute means 10,000 timestamps per key. It's fine for low limits and a bad idea for high ones.

Sliding window counter

This is a clever compromise. Keep counters for the current and previous fixed windows, then weight the previous one by how much of it still overlaps the sliding window:

estimate = current_count + previous_count × (1 − elapsed_fraction_of_current_window)

It smooths out the boundary burst using only two counters per key. The estimate assumes requests in the previous window were evenly spread, which is usually close enough.

Token bucket

Each key has a bucket holding up to capacity tokens. Tokens are added at a steady rate. Each request takes one token, and a request that finds the bucket empty is rejected.

The token bucket is my default because its two parameters map directly to what product people ask for: "on average 10 requests per second, but allow bursts of up to 50." The rate is the sustained limit and the capacity is the burst allowance. It also only needs two stored values per key: the token count and the last refill time.

A single server: don't write it yourself

In Go, golang.org/x/time/rate already implements a token bucket. rate.NewLimiter(r, b) creates one with a refill rate of r tokens per second and a burst of b. A per-client HTTP middleware looks like this:

type clientLimiters struct {
	mu       sync.Mutex
	limiters map[string]*rate.Limiter
	r        rate.Limit
	burst    int
}

func (c *clientLimiters) get(key string) *rate.Limiter {
	c.mu.Lock()
	defer c.mu.Unlock()
	l, ok := c.limiters[key]
	if !ok {
		l = rate.NewLimiter(c.r, c.burst)
		c.limiters[key] = l
	}
	return l
}

func (c *clientLimiters) Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("X-API-Key") // or the authenticated user ID
		if !c.get(key).Allow() {
			w.Header().Set("Retry-After", "1")
			http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
			return
		}
		next.ServeHTTP(w, r)
	})
}

This version never evicts idle clients, so the map grows forever. In real code, store a "last seen" time next to each limiter and sweep old entries periodically, or use an LRU. I left it out to keep the example short, but don't ship it without that.

The bigger problem is that this limiter lives in one process. Run six replicas behind a load balancer and each client effectively gets six times the limit. Once you scale horizontally, the state has to be shared.

Many servers: a token bucket in Redis

Redis is the usual choice for shared limiter state: it's fast, it already sits in most stacks, and it can run Lua scripts atomically. Atomicity is the key point. A read-modify-write done in separate commands lets two servers read the same token count and both allow a request. A Lua script runs as a single step that nothing else can interleave with.

-- KEYS[1]  bucket key, e.g. "rl:{api-key}"
-- ARGV[1]  capacity (max tokens)
-- ARGV[2]  refill rate (tokens per second)
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])

-- Use Redis's clock, not the app server's, so clock skew between
-- servers can't hand out extra tokens.
local t   = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local state  = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts     = tonumber(state[2]) or now

-- Refill based on elapsed time, capped at capacity.
tokens = math.min(capacity, tokens + math.max(0, now - ts) / 1000 * rate)

local allowed, retry_ms = 0, 0
if tokens >= 1 then
  tokens  = tokens - 1
  allowed = 1
else
  retry_ms = math.ceil((1 - tokens) / rate * 1000)
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- An idle bucket is full again after capacity/rate seconds, so drop it then.
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 1000))

return { allowed, retry_ms }

A few design choices are worth pointing out:

• Redis's clock via TIME. If each app server passed in its own timestamp, a server with a fast clock would refill buckets early. Calling TIME inside a script works on Redis 5 and later, where scripts are replicated by their effects rather than by re-running them.
• Expiry keeps memory bounded. A bucket that's been idle long enough is full again anyway, so deleting it loses nothing.
• Returning the retry delay. The client can be told exactly when to try again instead of guessing.

Calling it from Go with go-redis:

var tokenBucket = redis.NewScript(tokenBucketLua)

func (l *RedisLimiter) Allow(ctx context.Context, key string) (bool, time.Duration, error) {
	res, err := tokenBucket.Run(ctx, l.rdb, []string{"rl:" + key}, l.capacity, l.rate).Int64Slice()
	if err != nil {
		return false, 0, fmt.Errorf("rate limit script: %w", err)
	}
	return res[0] == 1, time.Duration(res[1]) * time.Millisecond, nil
}

Script.Run tries EVALSHA first and falls back to EVAL if Redis doesn't have the script cached, so you don't pay to send the script body on every call.

If you run Redis Cluster, every key a script touches must hash to the same slot. This script only touches one key, so it's fine. Keep that in mind if you later add a second key, like a global limit alongside the per-user one.

Tell clients what happened

A rate limiter is part of your API, so design its responses like the rest of the API:

• Status 429 Too Many Requests, defined in RFC 6585.
• A Retry-After header with the number of seconds to wait. Well-behaved clients and many HTTP libraries respect it.
• Remaining-quota headers if clients need to pace themselves. Many APIs use X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, and the IETF has been working on standard RateLimit header fields. Pick one convention and document it.

When the limiter fails

Redis will be unreachable at some point. Your middleware then has two options:

• Fail open: allow the request. The product stays up, but you're unprotected until Redis comes back.
• Fail closed: reject the request. You stay protected, but a Redis outage becomes a full outage.

For most user-facing APIs, I fail open and alert loudly, because a rate limiter exists to protect the service, not to become its single point of failure. For things like login attempts or SMS sending, where abuse costs real money or security, I fail closed. Whichever you pick, make it a deliberate, documented decision, not whatever happens when the error falls through.

A useful middle ground is falling back to the in-process rate.Limiter from earlier with a stricter per-replica limit. It's imprecise, but it's much better than no limit at all.

Summary

• Start from the requirement: who is limited, what's protected, and what happens on failure.
• Token bucket is a great default because it maps directly to "sustained rate plus burst."
• In a single process, use x/time/rate. Across a fleet, use an atomic Redis script using Redis's clock.
• Return 429 with Retry-After, and decide on purpose whether to fail open or closed.

Want to talk system design? Drop me a message on the contact page.