StoryboardAI Engineering

Tokens

The small pieces of text a model actually reads and writes.

What are tokens?

Before a model sees any text, the text is cut into tokens. A token is often a short word, part of a longer word, or a punctuation mark. In English, one token is about three quarters of a word.

Tokens are the unit for everything practical. A model can only take in a fixed number of them at once. You are billed per token read and per token written. And because a model writes one token after another, longer answers take longer.

Numbers, code and rare words split into more tokens than you would guess. The word waterproof becomes two tokens, not one.

How text becomes tokens before a model reads it.Step 1: Text arrives. Step 2: It is cut into pieces. Step 3: Each piece becomes a number. Step 4: The model reads the numbers.1Text arrives2It is cut intopieces3Each piece becomesa number4The model readsthe numbers
How text becomes tokens before a model reads it.

Why do tokens matter?

Tokens are the unit of everything that matters in practice. The price of a call is counted in tokens. The amount a model can read at once is a number of tokens. The time a reply takes depends on how many tokens are written. Engineers who think in tokens can predict cost, speed and limits before they build.

How do tokens work?

  1. Before a model reads text, a program called a tokenizer cuts it into pieces from a fixed list of about a hundred thousand.
  2. Common words are one piece. Rarer words are split into several, and so are long numbers and names.
  3. Each piece is replaced by its number in the list. The model only ever sees the numbers.
  4. The model's reply is produced the same way in reverse: it picks one number at a time, and each is turned back into text.

When do tokens need your attention?

Count them when

  • Estimating what a feature will cost at scale
  • A prompt is getting close to the model's limit
  • A reply is being cut off or is too slow

Do not worry when

  • You are sending a short question and reading a short answer
  • You are still working out whether the idea works at all

Where do tokens sit in a real system?

Counting happens at the provider, and the count is returned with every reply. Most teams also count on their own side before sending, using the provider's tokenizer, so they can trim a prompt that would not fit and record cost per request.

What goes wrong with tokens?

  • Treating a token as a word. In English a token is about three quarters of a word, and in many other languages less.
  • Forgetting that the reply is counted too, and usually costs several times more per token than the prompt.
  • Assuming two models count the same text the same way. Each family has its own tokenizer.

What does tokens look like in practice?

The sentence "What should I wear for a rainy hike?" is 9 tokens. A 60-page handbook is about 36,000. At $3 per million tokens read, sending that handbook with every question costs about 11 cents each time, which is why later chapters go to such lengths to send less.

Watch it happen

These stories show tokens at work, one step at a time.