How to Engineer Rate Limiting

Rate limiting is often introduced as an algorithm problem: should we use a token bucket, a sliding window, or a fixed-window counter? Those are useful implementation questions, but they start too far down the stack. In a production system, the more important questions come first:

• What finite resource or business allocation are we trying to protect?
• Who is consuming it?
• What should count as consumption?
• What behavior do we actually want to permit?
• Where should excess work be stopped?
• How strict does the guarantee need to be?
• What does the client need to know about that guarantee?

Think of a fort facing an advancing army. A moat, an outer wall, and a guarded inner gate protect different boundaries; the gate to the keep cannot protect the outer wall from being overwhelmed. Layered defenses work when each protects something specific and all relevant approaches are covered.

Now think of an office parking lot on a busy morning. The arriving drivers are legitimate, but spaces are finite. A boom barrier can control admission at the entrance, provided its decision reflects available parking capacity; merely counting cars per minute cannot guarantee an open space. Rate limiting is likewise about deciding which work can safely enter a constrained system, not just counting arrivals.

These analogies bring us back to first-principles thinking: identify the resource, consumer, unit of consumption, policy, enforcement boundary, and guarantee before choosing a token bucket, Redis, an API gateway, or a dedicated rate-limit service. This article develops that reasoning before the implementation details.

Why limit anything?

Every production system has finite capacity: CPU, memory, worker threads, database connections, downstream-service throughput, network bandwidth, or some combination of them. As offered load increases, there comes a point where accepting more work no longer increases useful throughput. Latency rises, queues grow, deadlines are missed, and retries may generate even more traffic.

Average latency can hide the point at which the system has already become unsafe. Consider an illustrative payment service with a 500 ms request deadline. Under normal load, it might have p50 = 80 ms, p95 = 120 ms, and p99 = 180 ms. During overload, the median might still appear tolerable at 140 ms while p95 rises to 620 ms, p99 reaches 1.4 s, queue depth grows from 100 to 4,000, and retries rise from 2% to 18%.

That is why capacity and limiter thresholds should be validated against queue depth, error and retry rates, and latency percentiles relative to client deadlines—not average latency alone.

Google’s SRE guidance treats overload handling as a fundamental production concern and recommends rejecting work when a service cannot safely sustain the offered load rather than allowing the entire service to collapse (Google SRE: Addressing Cascading Failures, Google SRE: Service Best Practices).

So the first-principles problem is not:

How do I count requests?

It is:

How do I control how much work is admitted to a finite resource so that the resource remains useful?

Rate limiting is one mechanism for doing that.

Rate limiting also solves allocation and fairness problems

Protecting infrastructure is not the only reason to limit traffic.

Suppose ten merchants share the same payment API fleet. Even if the fleet could survive a burst from one merchant, we would not want that merchant to dominate everyone else’s capacity. An illustrative starting proposal is 100 requests per second per merchant. That sounds like a reasonable fairness rule. Before adopting it, though, what would it actually protect?

A limit can therefore express several different business or system requirements:

• Capacity protection: prevent a service or dependency from becoming overloaded
• Fairness: prevent one consumer from exploiting shared resources
• Abuse protection: constrain pathological or hostile request patterns
• Cost control: prevent one workload from consuming disproportionate system resources
• Commercial allocation: enforce capacity associated with plans or entitlements
• Priority protection: preserve capacity for higher-value work

Stripe describes several of these motivations in its production API: unexpected user traffic, buggy integrations, deliberate abuse, low-priority work interfering with critical traffic, and incidents where the platform cannot safely serve all normal traffic.

GitHub similarly describes API limits as a way to prevent abuse and denial-of-service behavior while preserving API availability for other users.

The important mindset shift is:

Rate limiting is not fundamentally about returning HTTP 429. It is about allocating access to finite or controlled capacity.

Rate limiting is only one form of admission control

Before going further, distinguish the related forms of admission control. They may all reject work, but they constrain different things.

Rate limit

Controls how quickly requests of work may arrive. Say 100 requests / second

Quota

Controls how much work may occur over a longer allocation period, such as 1,000,000 requests / month.

Rate Limits and Quota might sound similar but AWS API Gateway distinguishes request throttling from longer-period usage-plan quotas (AWS: API Gateway request throttling).

Concurrency limit

Controls how much work may be active simultaneously, such as active requests <= 50.

This distinction matters for long-running work.

Stripe operates both rate and concurrency limiters. It distinguishes rate limits and concurrency limits, and its engineering write-up describes concurrency limiting as protection for expensive, long-lived requests.

Adaptive admission

Changes allowed concurrency or load according to observed system behavior rather than using one permanently configured number.

Envoy’s adaptive concurrency filter, for example, adjusts outstanding-request limits using latency measurements relative to an estimated baseline (Envoy: Adaptive Concurrency).

Load shedding

Rejects work because the system is currently unhealthy or saturated.

Google SRE points out an important limitation of ordinary rate limiting: a configured request rate does not necessarily reflect the system’s current health. During overload, separate load-shedding mechanisms may be required (Google SRE: Addressing Cascading Failures).

Stripe makes the same distinction operationally: its rate limiters control normal traffic allocation, while load shedders react to actual fleet conditions during incidents (Stripe Engineering: Scaling your API with rate limiters).

These mechanisms belong to the broader idea of admission control:

Given the system’s policy and current state, should this unit of work be allowed to enter?

The rest of this article focuses on rate limiting while keeping those other admission controls in view.

First question: what are we protecting?

Return to the proposed 100 requests per second per merchant limit. If it rejects requests exactly as intended, what have we protected?

Consider the Payment API path:

Several different resources might become the bottleneck:

• API service CPU & Memory
• Request workers/Pods
• Database connections
• Locks around a payment object
• Risk / Fraud Detection Service’s capacity
• Downstream processor throughput
• A merchant’s purchased allocation.

Protecting one of these resources does not necessarily protect the others. The per-merchant request limit might preserve fairness in the shared API fleet. Whether it also protects workers, the Risk Service, or the processor depends on which work reaches each resource and what exhausts it first. The endpoint, Redis, and a chosen algorithm do not answer that question.

Google SRE recommends understanding actual service breaking points through load testing precisely because the first exhausted resource is not always obvious from architecture diagrams alone (Google SRE: Service Best Practices).

The first question should therefore be:

What becomes scarce, unsafe, unfair, or economically undesirable as demand increases?

If the answer is unclear, the limiter design is premature.

Who is consuming the resource?

Once you know what you are protecting, decide whose usage should be accounted for independently. The consumer might be identified by:

Client IP address
User ID
API Key
Merchant
Tenant
Calling service
Target Endpoint
Requested Resource

or composite identities like:

Merchant + Endpoint
Caller Service + Destination service
Merchant + Payment Resource

This defines the fairness boundary.

RFC 6585 intentionally does not prescribe how a server identifies the subject behind a 429 Too Many Requests response. It notes that a server might count per resource, across a server, across multiple servers, or identify a user through credentials or other state.

Production systems reflect that flexibility.

Redis’s rate-limiter documentation uses keys scoped by application-defined identities such as users or IP addresses, and the same underlying pattern can be used with API keys, tenants, or composite application identities.

Cloudflare’s advanced rate-limiting system, which is integrated into its WAF, can filter out traffic not only client’s IP, which is hardly static, but fields like hostname, URI patterns, any of request headers such as User-Agent or advanced fields like Bot Score, event complexity of a GraphQL query.

What should we meter?

Our proposed 100 requests per second per merchant policy counts every request equally. That may be enough for merchant fairness or for operations with comparable costs, but does it tell us how much work the payment system performs? AWS API Gateway uses request-count-based token-bucket throttling with separately configurable sustained rate and burst capacity.

A request is only a proxy for work. GET /payments/123 and POST /payments each consume one unit of the merchant allowance, although the first may be a cheap lookup and the second may trigger validation, persistence, risk checks, processor communication, and asynchronous work. Counting them equally can preserve fairness without accurately reflecting demand on those downstream resources.

The better question is:

Which measurable quantity correlates most closely with the resource or allocation we are trying to control?

Possible units include:

1. requests
2. transactions
3. tokens
4. payload size
5. in-flight jobs
6. concurrent operations

Modern APIs commonly use multiple dimensions. Google’s Gemini API, for example, documents limits across requests per minute, tokens per minute, requests per day, and, for applicable image models, images per minute.

GitHub combines ordinary request counts with other controls including concurrent requests, endpoint points, CPU-related usage, and content-creation limits.

The distinction is worth keeping in mind:

One request is not necessarily one unit of resource consumption.

Weighted consumption

Sometimes the system still benefits from one common budget, but different operations should consume different amounts of it.

Start with measurement. If load tests show that p95 worker time per operation predicts saturation, an illustrative conversion into weights might be:

These numbers are illustrative, not recommended defaults. The weights are coarse policy approximations: they translate observed resource consumption into a shared budget that the limiter can enforce.

The validation loop matters. If the weighted budget is exhausted while the protected resource remains mostly idle, the weights or capacity are too conservative. If the resource saturates before the budget is consumed, they are too permissive. Revisit them as workloads and dependencies change.

Weighted accounting still separates two questions: “How expensive is this operation?” and “How does available budget replenish over time?”

The first is a metering problem; the second concerns the admission algorithm, which Part II develops. GitHub’s point-based secondary limits are a concrete example of assigning different request classes different effective costs.

Define the policy before choosing the algorithm

Once the resource, consumer, and metric are known, we can state the behavior we want without naming an algorithm.

Our starting request-rate proposal addresses merchant fairness, but expensive authorization work may need a separate constraint. For illustration, suppose the requirements are:

Each merchant may sustain roughly 100 API requests per second with bounded bursts. No merchant may have more than 20 expensive authorization operations active at once.

These numbers are illustrative, not recommended defaults. The two constraints govern different aspects of the same traffic: how quickly requests arrive and how much costly work remains in flight.

Start with sustained rate—how much traffic is acceptable over time, such as 100 operations / second—and burst tolerance: how much may arrive briefly above that rate? AWS API Gateway makes the distinction explicit with a token refill rate and a separate bucket capacity. AWS documents these values as best-effort targets, not strict request ceilings (AWS: API Gateway throttling, AWS: API Gateway request throttling).

A quota answers a longer-term question, such as whether the merchant has 10 million operations / month. Concurrency answers a different one: even below the request-rate limit, slow authorizations can accumulate. If active work is scarce, cap what remains in flight rather than only what arrives.

Scope

What does the configured policy apply to?

process
service fleet
region
global platform

Finally, state the strictness: is this a hard contractual ceiling or an approximate protective guardrail? Together, these requirements define what the algorithm must achieve. Token buckets, sliding windows, and GCRA are implementation choices for Part II.

How strict does the guarantee need to be?

The same merchant limit takes on a different meaning across three regions. If each region independently allows 100 requests per second per merchant, the deployment could offer that merchant 300 requests per second of configured sustained allowance, even before bursts. That may be acceptable if the policy is intentionally per region. If the intended promise is 100 requests per second per merchant across all regions, the regions need to coordinate consumption or divide a shared budget. The number alone did not tell us which guarantee we meant.

A shared limit also needs an outage rule. If the coordination system is unavailable, should the API reject work, or continue locally and risk temporary overshoot? The answer depends on what the policy promises and what the resource can tolerate.

For a protective guardrail configured at 1,000 RPS, a temporary 1,040 RPS might be harmless. For a hard contractual allocation or scarce downstream dependency, that same overshoot might be unacceptable.

AWS explicitly documents API Gateway throttling as best-effort rather than a guaranteed hard ceiling, which is a useful example of the distinction between a configured target and a strict distributed guarantee (AWS: API Gateway request throttling).

The implementation guarantee must also be separated from the customer-facing promise:

A best-effort guardrail should not silently become the mechanism for enforcing a contractual entitlement. Conversely, a commercial limit should not promise exact global accounting unless the production architecture can provide it.

A complete requirement therefore names the scope, acceptable overshoot, behavior during coordination failure, and recovery behavior. Those choices lead to three common failure postures.

Fail open

If the limiter cannot make a decision, allow the work.

Useful when:

• availability matters more than temporary policy violation
• the protected resource has enough safety margin
• a limiter outage should not become an application outage

Fail closed

If the limiter cannot make a decision, reject the work.

Useful when:

• violating the allocation is more dangerous than losing availability
• the downstream resource has a hard capacity boundary
• the policy has contractual or security significance

Fallback to Local Enforcement

Continue with a weaker local limit or a previously allocated local quota when the shared authority is unavailable. This preserves more availability than failing closed, but weakens the fleet-wide guarantee and can permit bounded overshoot.

Envoy’s Rate Limit Quota filter provides a concrete implementation: after an assignment expires, an Envoy instance can continue using the last assignment or apply a predefined fallback value. The filter therefore keeps the admission decision local during a control-plane failure, while accepting that the local view may no longer reflect the latest global allocation (Envoy: Rate Limit Quota).

The distributed implementation consequences of fail-open, fail-closed, and degraded local enforcement are examined in Part III. For now, the important point is:

Strictness is a business and architecture requirement, not a datastore property discovered later.

Define the API rate-limit contract

The guarantee also shapes what the API can tell callers. If one region rejects a merchant request while regions keep independent counts, its local counter cannot truthfully report a precise global remaining allowance. The response should at least say that the request was throttled; it may also answer:

Which policy was exceeded?
When should I retry?
What is my limit?
How much capacity remains?
When will capacity reset?
Is this value exact or advisory?

HTTP provides a standard status code for the first part.

RFC 6585 defines 429 Too Many Requests and allows the response to include Retry-After to indicate how long the caller should wait before trying again. Importantly, the RFC does not dictate the rate-limiting implementation itself (RFC 6585: 429 Too Many Requests, RFC 9110: Retry-After).

Real APIs often expose more.

Stripe returns a Stripe-Rate-Limited-Reason value that distinguishes categories including global rate, global concurrency, endpoint rate, endpoint concurrency, and resource-specific limiting (Stripe: Rate limits).

That tells the developer which capacity envelope the caller collided with, not just that a request failed.

For our hypothetical payment gateway, consider two possible response contracts. In the first, suppose the provider can identify a merchant-wide rate policy and offer meaningful, though advisory, retry timing:

HTTP/1.1 429 Too Many Requests
Retry-After: 2
RateLimit-Reason: merchant-global-rate
RateLimit-Policy-Scope: merchant-global
RateLimit-Precision: advisory

The caller can infer that it should wait at least two seconds before retrying. The provider-specific RateLimit-* fields explain which policy rejected the request and that the reported timing is advisory rather than an exact global counter.

The second is a concurrency rejection, which may deliberately omit Retry-After:

HTTP/1.1 429 Too Many Requests
RateLimit-Reason: payment-create-concurrency
RateLimit-Policy-Scope: merchant-payment-create
RateLimit-Precision: advisory

Here, capacity depends on currently executing requests, so the provider cannot truthfully name a fixed reset time. The client should use bounded exponential backoff with jitter rather than interpreting a missing Retry-After as permission to retry immediately.

RateLimit-Reason, RateLimit-Policy-Scope, and RateLimit-Precision are other provider-specific headers, not standardized HTTP fields. The two responses communicate different contracts.

An engineer’s takeaway is:

Define what the provider promises to communicate before deciding how the limiter computes it.

That promise constrains later decisions. A provider should not advertise an exact global remaining count or reset time if the underlying design can produce only a local, stale, or approximate view.

Where should enforcement happen?

With the policy and client promise defined, the next question is where to stop excess work. The central trade-off is:

Reject earlier to avoid more work; reject later to make the decision with richer application context.

A cooperative client can pace its own requests and avoid generating unnecessary traffic. This is useful as an optimization, but it cannot replace provider-side enforcement: buggy or hostile clients can ignore the published policy.

Edge or WAF

The edge is the earliest practical enforcement point for incoming traffic. Rejecting here prevents excess requests from reaching the origin and consuming deeper infrastructure.

This layer is well suited to coarse signals such as source address, hostname, path, request headers or known abuse patterns. It usually has less business context than the application. Cloudflare describes blocking matching traffic before it reaches the customer’s origin as an advantage of edge rate limiting (Cloudflare Engineering: Counting things a lot of different things). AWS similarly describes WAF rate-based rules as availability protection rather than precise request-rate accounting (AWS WAF: Rate-based rule caveats).

API gateway

The gateway commonly knows the caller and the requested route:

merchant
API key
route
HTTP method
product plan

It can therefore enforce consumer or API-level policies before requests reach application code. AWS API Gateway supports throttling at account, Region, route and method-related scopes (AWS: API Gateway throttling). Envoy provides both local and external-service-based rate-limit filters at the proxy layer (Envoy: Local Rate Limit, Envoy: Rate Limit).

Application or service boundary

Some decisions require context that the gateway does not have:

merchant tier
payment state
operation type
resource ownership
estimated work cost

The application service may therefore be the right enforcement point. The trade-off is that routing, authentication, parsing and other preliminary work has already occurred before a request is rejected.

Dependency boundary

Sometimes the constrained resource is deeper than the application itself.

Suppose the Payment Service can handle 10,000 requests per second, but the Risk Service can safely process only 2,000 expensive evaluations per second. Limiting all payment requests would unnecessarily reject operations that do not use the Risk Service. A control immediately before the Risk Service can target the actual constrained work.

This gives us a practical rule:

Place enforcement close enough to understand the resource, but early enough to avoid unnecessary work.

Layer your enforcement controls

The fort analogy now has a concrete counterpart: a payment request crosses several independently constrained resources—edge capacity, shared API capacity, payment workers, and downstream processor capacity. Each control in the following table addresses a different boundary.

These controls are not duplicates when they protect different resources or enforce different allocations. Envoy, for example, supports using local rate limiting together with global rate limiting (Envoy: Local Rate Limit).

One request may cross several capacity boundaries. Add an enforcement control where it enters each independently constrained resource.

Protect every traffic path to the resource

After selecting those controls, check their coverage. A per-merchant control at the API Gateway can govern public merchant requests perfectly yet leave the Risk Service exposed to work that never passes through the gateway.

Trace the public path:

External client → API Gateway → Payment Service → Risk Service

Now consider an internal batch worker that reaches the same dependency directly:

Batch Worker → Risk Service

The batch worker bypasses the gateway and can still overload the Risk Service. A correctly enforced merchant limit is therefore insufficient protection for this particular dependency.

External-to-platform traffic is often called north–south traffic. Calls between internal services are called east–west traffic. The important question is not the terminology, but whether any path can reach the protected resource without crossing an appropriate control.

Either place one limiter at a boundary shared by all relevant paths or place separate controls on the individual paths. Google SRE similarly recommends rejecting work upstream of an overloaded system where possible, rather than forcing the constrained dependency to absorb all excess work (Google SRE: Addressing Cascading Failures).

A limiter protects only the traffic that passes through it. Cover every path that can exhaust the resource.

How should consumption be tracked across the system?

Selecting an enforcement point does not yet tell us how multiple enforcers will honor the same allowance. To decide whether to admit the next request, a limiter needs the policy and enough information about previous or currently active consumption.

What must the limiter remember?

We will call this remembered information limiter state.

consumer identity
+
proposed consumption
+
policy
+
remembered consumption
→
admission decision

The exact state depends on the policy and the algorithm used to enforce it:

These examples preview the relationship between algorithms and state; Part II develops it. The architectural concern here is which decision-makers may maintain independent consumption records and which must share a common view.

Limiter state is the remembered consumption information required to make the next admission decision.

Which enforcers need the same consumption view?

The enforcement point is where a request is admitted or rejected. The state authority is where the consumption information needed for that decision is maintained. Several enforcement points may consult one authority, while one enforcement point may apply several policies backed by different authorities.

Two scopes therefore need to be distinguished:

For example, if two gateway processes each independently apply 100 requests per second per merchant, they could together provide 200 requests per second of configured sustained allowance for one merchant, even before bursts. The business scope is still the merchant, but the coordination scope is each process. If the intended allowance is shared across the fleet, those enforcers must share a consumption view. The same question applies when the fleet spans regions.

Which enforcement decisions must share consumption information for the promised limit to remain meaningful?

Common ways to organize limiter state

The following models are an architectural synthesis rather than an industry-standard classification. They help compare who owns the consumption view and what coordination each model requires.

Regional and global are coordination scopes, not separate state-organization models. A shared or sharded authority may operate within one region or span several regions, with different latency, availability and consistency consequences.

Choose the smallest coordination scope that can make the promised limit sufficiently true.

Technology follows the state requirements

Independent local state may live in process memory. Shared or sharded state might eventually be implemented using Redis, a database, a cache or a dedicated rate-limit service. These technologies are not interchangeable, and the state-organization model alone is not enough to choose among them.

The eventual choice must support the algorithm’s state representation and atomic transitions, the required coordination scope, decision latency, availability and fallback behavior, limiter-key distribution, durability requirements and operational cost. Redis, for example, documents counter-based limiters and Lua scripting for atomic read-decide-update operations (Redis: Rate Limiter). Envoy’s global rate-limit architecture instead lets a proxy delegate matching decisions to an external service (Envoy: Rate Limit, Envoy Rate Limit Service API).

Part II defines the algorithms and their logical state transitions. Part III examines how those transitions can be implemented and coordinated using local memory, shared datastores and dedicated services.

First determine who must share consumption state. Choose the algorithm next, and only then choose a technology capable of implementing its state transitions at the required scope.

Economic proportionality

The limiter is infrastructure too, with its own latency, spending, failure modes, and on-call burden. A multi-region service with a highly available datastore and cross-region coordination may be disproportionate for a non-critical operation that can tolerate, say, 5% temporary overshoot. In that case, the control could cost more—and introduce more fragility—than the problem it solves.

The reverse mistake is to use a cheap approximate limit for an extremely scarce or contractually constrained resource. In our payment gateway, a shared cross-region processor allocation may justify coordination that a process-local worker guardrail does not.

Is the cost and operational complexity of the limiter proportional to the value and strictness of the guarantee?

Compare the guarantee with the latency, infrastructure spend, availability coupling, operational toil, and implementation complexity needed to provide it. Stronger is better only when the requirement warrants the cost.

Putting it together: a multi-tenant payment gateway

Our 100 requests per second per merchant proposal remains a plausible fairness control at the API Gateway, if measurement supports the threshold and the instances coordinate at the promised scope. But it cannot, by itself, protect every part of the payment system. Stripe likewise documents distinct global, endpoint, resource-specific, and concurrency limits in its API (Stripe: Rate limits). The design becomes clearer when each missed constraint is paired with its own decision:

These are candidates, not defaults. For each constrained resource, ask: who consumes it, what work should count, and which traffic paths reach it—including batch work? Place the control where it can stop that work; decide which enforcers must share consumption state; specify overshoot and failure behavior before making a client promise. Finally, test whether the resource remains safe under load and whether the control’s cost is justified.

Rate limiting does not make payment retries safe on its own. Mutation retries also need an idempotency contract, as documented by Stripe and Adyen.

The merchant limit survives, but as one fairness control within a set of policies tied to distinct consumers and capacity boundaries. Part II examines the algorithms behind those policies; Part III examines how to coordinate their state across production infrastructure.