StoryboardAI Engineering

Retries and backoff

Trying a failed request again after a wait that grows each time, with some randomness.

What are retries and backoff?

Many failures are temporary: a rate limit, a brief overload, a dropped connection. The simple cure is to try again. Done carelessly, that makes things worse, because thousands of clients retrying at the same instant become a second wave of traffic.

So each retry waits longer than the last, for example one second, then two, then four. This is called exponential backoff. A little randomness is added to each wait so that clients do not all come back together.

Two limits keep it sane. A timeout gives up on any single attempt that takes too long. And after several failures in a row, the client stops trying for a while instead of hammering a service that is clearly down.

Each retry waits longer than the one before.Step 1: A call fails. Step 2: Wait 1 second, try again. Step 3: Wait 2, then 4. Step 4: Succeed, or stop and report.1A call fails2Wait 1 second, tryagain3Wait 2, then 44Succeed, or stopand report
Each retry waits longer than the one before.

Why do retries and backoff matter?

Calls over the internet fail for reasons that pass in seconds: a brief overload, a dropped connection, a timeout. Trying again usually works. Trying again badly, at once and without limit, turns a small problem into an outage. Retrying well is a basic part of making an AI product dependable.

How do retries and backoff work?

  1. A call fails with an error that is likely to pass, such as a timeout or an overload.
  2. The system waits a short time and tries again.
  3. Each further failure doubles the wait, with a little randomness added.
  4. After a fixed number of tries, it stops and reports the failure, or switches to a fallback.

When should you use retries and backoff?

Use it when

  • A call failed because of a timeout, an overload or a dropped connection
  • The action is safe to repeat

Think twice when

  • The error says the request itself is wrong. Repeating it will fail the same way
  • The action is not safe to repeat and has no key to prevent a double

Where do retries and backoff sit in a real system?

Retries live in the code that makes outside calls: to the model, to tools, to other services. They sit beneath the harness logic, so every call benefits without each one being written specially.

What goes wrong with retries and backoff?

  • Retrying immediately and for ever.
  • Retrying an action that moves money without a key that makes it safe to repeat.
  • Every worker retrying at the same instant, creating a fresh surge.

What does retries and backoff look like in practice?

A provider returns "overloaded". The system waits 1 second and tries again, then 2, then 4. The third try succeeds. The customer saw a slightly slower reply. Without the waits, thousands of clients retrying together would have kept the provider down.

Watch it happen

These stories show retries and backoff at work, one step at a time.