StoryboardAI Engineering

Chance and temperature

The model picks each token by weighted chance. Temperature sets how adventurous it is.

What is sampling and temperature?

For every next token, a model produces odds for each possibility. It then picks one at random, weighted by those odds. That is why the same question can get different answers.

Temperature is a dial on that choice. At zero, the model always takes the most likely token and its output is nearly the same every time. Higher values give less likely tokens more of a chance, which reads as more varied and creative.

Use a low temperature when code will read the output or when facts matter. Use a higher one for brainstorming and writing where variety helps.

Each token is a weighted roll of the dice.Step 1: The model scores every next token. Step 2: Scores become odds. Step 3: One is picked by chance. Step 4: Repeat for the next token. Then it repeats from step 1.1The model scoresevery next token2Scores become odds3One is picked bychance4Repeat for thenext tokenand round again
Each token is a weighted roll of the dice.

Why does sampling and temperature matter?

Ask a model the same question twice and you may get two different answers. That is by design, and it surprises anyone used to software that behaves the same way every time. Understanding where the variation comes from tells you when to turn it down, when to welcome it, and why tests must be run more than once.

How does sampling and temperature work?

  1. For each position in its reply, the model works out a score for every possible next token.
  2. The scores become odds: one token might have a 60% chance, another 25%.
  3. The model picks one at random according to those odds, like a weighted dice roll.
  4. A setting called temperature changes the odds. Low makes the likeliest token win almost always. High spreads the chances out.

When should you change the temperature?

Lower it when

  • Filling in a form or extracting facts
  • Choosing a tool or a category
  • You want runs to be as repeatable as possible

Raise it when

  • Writing that should feel fresh, such as product descriptions
  • Brainstorming many different options

Where does sampling and temperature sit in a real system?

Temperature is a number sent with each model call. It can differ per task inside one product: near zero for the step that sorts a message, higher for the step that drafts a friendly reply.

What goes wrong with sampling and temperature?

  • Expecting a temperature of zero to make output identical every time. It makes it nearly so, not exactly.
  • Testing a prompt once and trusting the result.
  • Raising temperature to fix dull writing when the real problem is a vague prompt.

What does sampling and temperature look like in practice?

After "The weather today is", a model gives "sunny" 41% odds, "cloudy" 22% and "lovely" 9%. At low temperature it writes "sunny" nearly every time. At high temperature "lovely" turns up one time in six, and the rest of the sentence changes with it.

Watch it happen

These stories show chance and temperature at work, one step at a time.