StoryboardAI Engineering

Code execution

The model writes a small program and a computer runs it, so exact work is done exactly.

What is code execution?

A model predicts text. That is a poor way to add up two hundred numbers, sort a table or count rows, and it gets worse as the data grows.

So instead of answering, the model writes a short program that would produce the answer. Your code runs it and returns the output. The arithmetic is done by a computer, which does not make arithmetic mistakes.

When the program fails, the error message goes back to the model, which reads it, fixes the code and tries again. The model handles the reasoning; the computer handles the exactness.

The model writes the code. A computer does the arithmetic.Step 1: A question needs exact numbers. Step 2: The model writes a short program. Step 3: A sandbox runs it. Step 4: The model reads the result.1A question needsexact numbers2The model writes ashort program3A sandbox runs it4The model readsthe result
The model writes the code. A computer does the arithmetic.

Why does code execution matter?

Models are unreliable at arithmetic and at handling large amounts of data, and they state wrong totals as confidently as right ones. They are, however, good at writing code. Letting a model write a short program and having a computer run it gives exact answers for sums, tables and analysis, with working that can be inspected.

How does code execution work?

  1. The model is given a tool that runs code and returns whatever it prints.
  2. Faced with a calculation, it writes a short program and asks for it to be run.
  3. The harness runs that program in a sealed environment and captures the output.
  4. The output is returned to the model, which reads it and writes the answer.

When should you use code execution?

Use it when

  • An answer depends on arithmetic or statistics
  • Data needs sorting, filtering or charting
  • A claim can be settled by trying it, such as whether code passes a test

Think twice when

  • There is no safe, sealed place to run the code
  • The answer is a matter of judgement, not calculation

Where does code execution sit in a real system?

The code runs in a sandbox operated by your harness or by the model's provider. It must be separate from your real systems, because the program was written by a model and may do anything.

What goes wrong with code execution?

  • Running model-written code on a machine that holds real data or secrets.
  • Letting the model quote a number it worked out in its head when it could have run the sum.
  • Returning huge outputs to the model without trimming them.

What does code execution look like in practice?

A customer uploads a year of orders and asks what they spent on boots. The model writes eight lines that load the file, keep the rows for boots and add the totals. The sandbox runs them and prints 412.50. The model reports that figure, and the code is kept as the working.

Watch it happen

These stories show code execution at work, one step at a time.