Launch day: AI Engineering, story 31
The shop's biggest sale begins, traffic jumps tenfold and the model provider stumbles. Watch HM Nayem's message survive it.
How one request survives rate limits, timeouts and a provider outage.
The story, step by step
The busiest hour of the year
The shop's biggest sale has just opened, and ten times the usual number of people are in the chat. HM Nayem is one of them, with a simple question. This is the hour when everything that can go wrong does.
A limit nobody thought about
A model provider shares its computers between many customers. To be fair to all, each has a cap on requests per minute, called a rate limit. On a normal day the shop is nowhere near it. Today it is far over.
The model says no
HM Nayem's request arrives and is turned away. Nothing is wrong with the question. The provider is simply saying: too many, come back in a moment. If the shop's code does nothing about it, HM Nayem sees an error.
First fix: try again, politely
A refusal like this is temporary, so the answer is to try again. But if nine thousand clients all retry at the same instant, that is a second wave. So each one waits, a bit longer every time, with a little randomness so they do not return together.
Second try, and it works
A second later the gateway sends HM Nayem's request again. This time there is room, and the reply comes back. HM Nayem waited one extra second and saw nothing unusual.
Then the provider itself stumbles
Twenty minutes later it gets worse. The main provider has trouble of its own and stops answering for a few minutes. This is not a polite “come back soon”. Requests go out and nothing comes back at all.
Do not wait forever
Two rules stop a bad moment turning into a pile-up. A timeout: give up on any attempt that takes too long. And a breaker: after several failures in a row, stop sending to that provider for a while instead of hammering something that is clearly down.
A second road
With the main model paused, the gateway sends requests to a backup: a similar model from a different provider. This is called a fallback. The answers may be worded a little differently. They arrive, which is what counts today.
The backup was tested weeks ago
A fallback only works if it was prepared. Two models respond differently to the same prompt. So the shop ran its eval set against the backup long before today, and adjusted the prompt until it passed.
One place for all of this
Retries, waits, timeouts, the breaker, the fallback. The shop has a chat assistant, a staff tool and nightly reports, and none of them should each build this again. So all of it lives in one service that every model call goes through: the gateway.
Who goes first
When there is not enough room for everyone, the gateway also decides the order. A customer waiting in a chat goes ahead of a staff tool, and both go ahead of tonight's reports, which can simply wait.
If nothing works at all
And if both providers were down? Then the product should say so plainly and still be useful, instead of spinning forever. An honest message and a next step beat a blank screen.
What HM Nayem saw
Back to today. Through a rate limit and an outage, HM Nayem saw one reply arrive a second late. Tens of thousands of requests failed at the provider. A dozen failed for a customer.
Now at real speed
Refused, wait, retry. Down, switch, answer. Model providers will fail from time to time, and that is normal. The work is making sure the product does not fail with them.
Providers will fail. The product should not.
- A rate limit caps requests per minute, and a traffic spike finds it.
- Temporary refusals are retried after a wait that grows each time.
- Timeouts and a breaker stop a failure turning into a pile-up.
- A tested fallback model takes over when the main one is down.
- A gateway puts all of this in one place for every team.
Ideas in this story
- Rate limits: A cap the provider sets on how many requests or tokens you may send each minute.
- Retries and backoff: Trying a failed request again after a wait that grows each time, with some randomness.
- Fallbacks: A second model or provider that takes over when the first one is unavailable.
- The model gateway: One internal service that every model call passes through, handling keys, limits, retries and fallbacks in one place.