What are rate limits?
A model provider shares its computers among many customers. To keep one from crowding out the rest, each customer has a limit on requests and tokens per minute.
Go over it and the provider refuses the extra requests with an error that says, in effect, too many, try again shortly. Nothing is wrong with the request itself.
Limits are invisible until the day traffic spikes, which is usually the day it matters most. Teams watch how close they run, ask for higher limits before a launch, and build in ways to cope when the limit is hit anyway.
Why do rate limits matter?
Every model provider caps how many requests and tokens a customer may send per minute. Cross the cap and requests are refused. It tends to happen at the worst moment, when traffic is highest. A product that has not planned for the limit will fail exactly when it is most used.
How do rate limits work?
- The provider counts your requests and tokens in each minute.
- A request over the limit is refused, with a message saying how long to wait.
- Your system keeps its own count and holds requests back before the limit is reached.
- Requests that must wait go into a queue and are sent as room appears.
- When a refusal does arrive, everything slows down briefly before trying again.
When do rate limits need planning for?
Plan for them when
- Traffic comes in bursts
- Many workers or agents share one account
- A launch or a sale will multiply usage
They rarely matter when
- Usage is low and steady
- You are a long way under your limit
Where do rate limits sit in a real system?
The limit is enforced by the provider. The planning lives in your own system, usually in one shared place that every model call passes through, so that all workers draw from the same allowance.
What goes wrong with rate limits?
- Retrying at once when refused, which makes the overload worse.
- Letting each worker keep its own count, so together they overshoot.
- Discovering the limit in production on the busiest day of the year.
What does rate limits look like in practice?
Twenty workers want to make 1,100 calls a minute against a limit of 400. A shared allowance hands out 380 a minute. No call is refused, and the night's work takes a little longer.