StoryboardAI Engineering

The model gateway

One internal service that every model call passes through, handling keys, limits, retries and fallbacks in one place.

What is a model gateway?

A company soon has many programs calling models: the chat assistant, a staff tool, nightly reports. If each handles keys, retries and fallbacks its own way, every fix has to be made many times and nobody can see the whole picture.

A gateway is a single door in front of all model providers. Every call goes through it. It holds the secret keys, applies budgets and limits per team, retries and falls back, and records what was used.

It also decides priority when capacity is short: a live customer conversation goes ahead of a report nobody is waiting for.

One door for every model call.Step 1: Your services send requests. Step 2: The gateway adds keys and rules. Step 3: It forwards to a provider. Step 4: It records cost and timing.1Your services sendrequests2The gateway addskeys and rules3It forwards to aprovider4It records costand timing
One door for every model call.

Why does a model gateway matter?

As a product grows, model calls are made from many places, each with its own handling of keys, limits, retries and logging. Changes become risky and costs become invisible. A gateway is one door that every model call goes through, so those concerns are handled once and consistently.

How does a model gateway work?

  1. Every part of your system sends model requests to the gateway, not to providers directly.
  2. The gateway holds the provider keys and adds them to each request.
  3. It applies shared rules: limits, retries, fallbacks, routing.
  4. It records every call with its cost, timing and outcome.
  5. It forwards the request to the chosen provider and passes the reply back.

When should you use a model gateway?

Use it when

  • Several services or teams call models
  • You use more than one model or provider
  • You need one place to see cost and set limits

Think twice when

  • One small program makes all the calls. A gateway would be an extra moving part

Where does a model gateway sit in a real system?

It sits between your own services and the model providers. Because everything passes through it, it is also the natural place for caching, budgets per team and switching models without changing application code.

What goes wrong with a model gateway?

  • Making it a single point of failure with no spare.
  • Letting teams bypass it with their own keys.
  • Logging full prompts through it without masking private data.

What does a model gateway look like in practice?

During a provider outage, a team changes one setting in the gateway and every service switches to a second provider within seconds. The same gateway shows that one feature accounts for 60% of the month's bill.

Watch it happen

These stories show the model gateway at work, one step at a time.