AI Engineering, story 31Launch day

The busiest hour of the year

1 of 14

Launch day: AI Engineering, story 31

The shop's biggest sale begins, traffic jumps tenfold and the model provider stumbles. Watch HM Nayem's message survive it.

How one request survives rate limits, timeouts and a provider outage.

The story, step by step

  1. The busiest hour of the year

    The shop's biggest sale has just opened, and ten times the usual number of people are in the chat. HM Nayem is one of them, with a simple question. This is the hour when everything that can go wrong does.

  2. A limit nobody thought about

    A model provider shares its computers between many customers. To be fair to all, each has a cap on requests per minute, called a rate limit. On a normal day the shop is nowhere near it. Today it is far over.

  3. The model says no

    HM Nayem's request arrives and is turned away. Nothing is wrong with the question. The provider is simply saying: too many, come back in a moment. If the shop's code does nothing about it, HM Nayem sees an error.

  4. First fix: try again, politely

    A refusal like this is temporary, so the answer is to try again. But if nine thousand clients all retry at the same instant, that is a second wave. So each one waits, a bit longer every time, with a little randomness so they do not return together.

  5. Second try, and it works

    A second later the gateway sends HM Nayem's request again. This time there is room, and the reply comes back. HM Nayem waited one extra second and saw nothing unusual.

  6. Then the provider itself stumbles

    Twenty minutes later it gets worse. The main provider has trouble of its own and stops answering for a few minutes. This is not a polite “come back soon”. Requests go out and nothing comes back at all.

  7. Do not wait forever

    Two rules stop a bad moment turning into a pile-up. A timeout: give up on any attempt that takes too long. And a breaker: after several failures in a row, stop sending to that provider for a while instead of hammering something that is clearly down.

  8. A second road

    With the main model paused, the gateway sends requests to a backup: a similar model from a different provider. This is called a fallback. The answers may be worded a little differently. They arrive, which is what counts today.

  9. The backup was tested weeks ago

    A fallback only works if it was prepared. Two models respond differently to the same prompt. So the shop ran its eval set against the backup long before today, and adjusted the prompt until it passed.

  10. One place for all of this

    Retries, waits, timeouts, the breaker, the fallback. The shop has a chat assistant, a staff tool and nightly reports, and none of them should each build this again. So all of it lives in one service that every model call goes through: the gateway.

  11. Who goes first

    When there is not enough room for everyone, the gateway also decides the order. A customer waiting in a chat goes ahead of a staff tool, and both go ahead of tonight's reports, which can simply wait.

  12. If nothing works at all

    And if both providers were down? Then the product should say so plainly and still be useful, instead of spinning forever. An honest message and a next step beat a blank screen.

  13. What HM Nayem saw

    Back to today. Through a rate limit and an outage, HM Nayem saw one reply arrive a second late. Tens of thousands of requests failed at the provider. A dozen failed for a customer.

  14. Now at real speed

    Refused, wait, retry. Down, switch, answer. Model providers will fail from time to time, and that is normal. The work is making sure the product does not fail with them.

Providers will fail. The product should not.

Ideas in this story