What are retries and backoff?
Many failures are temporary: a rate limit, a brief overload, a dropped connection. The simple cure is to try again. Done carelessly, that makes things worse, because thousands of clients retrying at the same instant become a second wave of traffic.
So each retry waits longer than the last, for example one second, then two, then four. This is called exponential backoff. A little randomness is added to each wait so that clients do not all come back together.
Two limits keep it sane. A timeout gives up on any single attempt that takes too long. And after several failures in a row, the client stops trying for a while instead of hammering a service that is clearly down.
Why do retries and backoff matter?
Calls over the internet fail for reasons that pass in seconds: a brief overload, a dropped connection, a timeout. Trying again usually works. Trying again badly, at once and without limit, turns a small problem into an outage. Retrying well is a basic part of making an AI product dependable.
How do retries and backoff work?
- A call fails with an error that is likely to pass, such as a timeout or an overload.
- The system waits a short time and tries again.
- Each further failure doubles the wait, with a little randomness added.
- After a fixed number of tries, it stops and reports the failure, or switches to a fallback.
When should you use retries and backoff?
Use it when
- A call failed because of a timeout, an overload or a dropped connection
- The action is safe to repeat
Think twice when
- The error says the request itself is wrong. Repeating it will fail the same way
- The action is not safe to repeat and has no key to prevent a double
Where do retries and backoff sit in a real system?
Retries live in the code that makes outside calls: to the model, to tools, to other services. They sit beneath the harness logic, so every call benefits without each one being written specially.
What goes wrong with retries and backoff?
- Retrying immediately and for ever.
- Retrying an action that moves money without a key that makes it safe to repeat.
- Every worker retrying at the same instant, creating a fresh surge.
What does retries and backoff look like in practice?
A provider returns "overloaded". The system waits 1 second and tries again, then 2, then 4. The third try succeeds. The customer saw a slightly slower reply. Without the waits, thousands of clients retrying together would have kept the provider down.