StoryboardAI Engineering

Batch jobs

Sending work that nobody is waiting for all at once, for results within hours at about half the price.

What is a batch job?

Not every model call has a person watching the screen. Tagging yesterday's conversations, summarising a backlog of documents, or grading an eval set can all finish any time before morning.

Providers offer a batch service for this. You hand over thousands of requests in one go and collect the results later, typically within a day. In exchange for giving up speed, the price is about half.

The habit worth building is asking of every piece of work: does anyone need this now? If not, it does not belong in the fast, expensive lane.

Two lanes for the same work.One at a time: Answer in seconds; Full price; For when someone is waiting. As a batch: Answer within hours; About half price; For work that can wait.One at a timeAnswer in secondsFull priceFor when someone is waitingAs a batchAnswer within hoursAbout half priceFor work that can wait
Two lanes for the same work.

Why does a batch job matter?

Not every request has a person waiting. Sorting last night's mail, reading ten thousand invoices, or labelling a dataset can finish any time in the next few hours. Providers offer a slower lane for exactly this, at about half the price. Using it for work that can wait is one of the simplest savings available.

How does a batch job work?

  1. Collect the requests that do not need an immediate answer.
  2. Send them to the provider together, as one batch.
  3. The provider works through them when it has spare capacity, usually within hours.
  4. Collect all the replies, and match each to its request.

When should you use a batch job?

Use it when

  • Nobody is waiting for the result
  • There are hundreds or thousands of similar requests
  • The work runs on a schedule, such as overnight

Think twice when

  • A person is waiting
  • Each request depends on the answer to the previous one

Where does a batch job sit in a real system?

Batches are sent by a background job, not by the part of your system that serves people. Results come back to the same job, which stores them and handles any that failed.

What goes wrong with a batch job?

  • Using the fast lane out of habit for work that could wait.
  • Forgetting that some requests in a batch can fail, and need to be found and retried.
  • Putting user-facing requests in a batch and wondering why they take hours.

What does a batch job look like in practice?

Ten thousand invoices need a first reading. Sent one by one at full speed, the reading costs $62. Sent as one batch at eleven at night, it costs $31 and is finished by 02:40. Nobody needed it sooner.

Watch it happen

These stories show batch jobs at work, one step at a time.