AI Engineering, story 1Hello

HM Nayem asks a question

1 of 14

Hello: AI Engineering, story 1

HM Nayem asks a shop's chat a simple question. Follow the message to the AI and watch the reply come back one piece at a time.

A message goes to the shop's server, then the model, and the reply streams back token by token.

The story, step by step

  1. HM Nayem asks a question

    HM Nayem opens a shop's chat and asks what to wear for a rainy hike. A reply will start appearing in under a second. Let's see where that message goes.

  2. It goes to the shop first

    The message does not go straight to the AI. It goes to the shop's own server first. Every AI product is built this way, and the next step shows the main reason.

  3. The server holds the key

    Using a model costs money, and the company that runs it needs to know who to bill. So the shop has a secret key, like a password. It lives only on the server. If it sat inside the app on a phone, anyone could copy it and spend the shop's money.

  4. The server writes a request

    The server wraps the question in a small request: which model to use, a limit on how long the reply may be, and the message itself. That is all a model call is.

  5. Off to the model

    The request travels over the internet to the company that runs the model. It is an ordinary web request, the same kind your browser makes all day.

  6. What is waiting at the other end

    A model is a very large program that learned the patterns of language by reading an enormous amount of text. It runs on powerful computers owned by an AI company. The shop does not own one. It rents it, one request at a time.

  7. Text becomes tokens

    The model does not read letters or words. The question is cut into small pieces called tokens. Tokens are also how usage is counted and billed.

  8. One token at a time

    Now it writes. A model does one thing only: guess the next token. It picks one, adds it to the text, and guesses again. A whole reply is that, repeated.

  9. Pick, add, repeat

    The chosen token is added to the end of the text, and the model looks at everything again to guess the one after. One round for every token. It never plans the whole sentence ahead. The reply builds up piece by piece.

  10. Knowing when to stop

    How does it know the reply is finished? One of the tokens it can pick is a special one that means “the end”. When that token wins, the model stops. If it never comes, the length limit the server set cuts the reply off instead.

  11. Each token is sent at once

    Each token is sent back the moment it exists, through the server and on to HM Nayem's phone. This is called streaming. It is why the reply seems to type itself.

  12. Why not wait for all of it?

    The model takes just as long either way. But with streaming, HM Nayem is reading after half a second instead of watching dots for three.

  13. What it cost

    When the reply ends, the model reports how many tokens it read and how many it wrote. That is the bill. The shop pays a tiny amount for each, and writing costs more than reading. For this one question: a small fraction of a cent.

  14. Now at real speed

    Here it is again, as fast as it really happens. One request out, a stream of tokens back. Every AI product you have used starts with exactly this.

That is one model call.

Ideas in this story