StoryboardAI Engineering

Fallbacks

A second model or provider that takes over when the first one is unavailable.

What is a fallback?

Retrying helps with a hiccup. It does nothing when a provider is down for ten minutes. For that, a product needs somewhere else to send the request.

A fallback is that second road: another model, another region or another provider, used automatically when the first keeps failing. The customer gets an answer, possibly a slightly different one, instead of an error.

It only works if it was prepared in advance. Different models respond differently to the same prompt, so the fallback has to be tested against the eval set before the day it is needed.

A prepared second choice for when the first one fails.Step 1: The first choice fails or is slow. Step 2: A trigger fires. Step 3: Switch to the second choice. Step 4: Record it, and switch back later.1The first choicefails or is slow2A trigger fires3Switch to thesecond choice4Record it, andswitch back later
A prepared second choice for when the first one fails.

Why does a fallback matter?

Model providers have outages and slow periods like any other service. A product that depends on a single model stops when that model does. A fallback is a prepared second choice, so that a failure upstream becomes a small drop in quality and not a blank screen.

How does a fallback work?

  1. Decide in advance what to use if the first choice fails: another model, another provider, a simpler answer.
  2. Set a clear trigger, such as two failed tries or no first word within a set time.
  3. When the trigger fires, switch automatically.
  4. Record that the fallback was used, and switch back when the first choice recovers.

When should you use a fallback?

Use it when

  • People depend on the product being up
  • A slow reply is as bad as none, as in voice
  • One provider's outage would otherwise stop everything

Think twice when

  • The fallback has not been tested. An untested fallback is a second failure
  • The work can simply wait and be retried later

Where does a fallback sit in a real system?

Fallbacks are configured where model calls are made, often in a gateway. Each stage of a product can have its own: a second voice, a second speech model, a person.

What goes wrong with a fallback?

  • Using prompts tuned for one model on a different one without testing them.
  • Never exercising the fallback until the day it is needed.
  • Falling back silently, so nobody knows quality has dropped.

What does a fallback look like in practice?

On a sale day a provider slows down. Replies that do not start within 0.6 seconds are sent to a smaller model from a second provider. The slowest one in twenty falls from 2.4 seconds to 1.1, and customers notice nothing.

Watch it happen

These stories show fallbacks at work, one step at a time.