AI Engineering, story 8Train it, or tell it?

"Train it on our data"

1 of 13

Train it, or tell it?: AI Engineering, story 8

The shop's owner wants a model trained on the shop's own data. See the three ways to change what a model does, and why training is usually the last one to reach for.

A model's behaviour is changed by telling, showing or training. Training suits style and narrow jobs at scale, and is wrong for facts that change.

The story, step by step

  1. "Train it on our data"

    The shop's owner has read that companies train their own AI. She asks HM Nayem to train one on everything the shop has. It sounds like the obvious way to make an assistant that knows the shop. It is one of three ways to change what a model does, and usually the last one worth trying.

  2. Three ways to change a model's behaviour

    You can tell it: put instructions in the prompt. You can show it: fetch the right text and place it in the prompt at the moment of asking. Or you can train it: change the model itself. The first two leave the model untouched and change what it reads. Only the third changes what it is.

  3. Telling

    The owner wants replies to be warm and short. HM Nayem adds two sentences to the instructions and three example replies. It takes ten minutes, and the next reply is warm and short. If it is wrong, it is changed back in ten more. For how a model should behave, telling is nearly always enough.

  4. Showing

    The owner also wants the assistant to know the returns policy, the prices and the opening hours. Those are facts, and facts change. So they are not put into the model at all. They are fetched from the help pages each time someone asks, as earlier stories showed. When a price changes, the page changes, and the very next answer is right.

  5. Training

    Training is different in kind. HM Nayem would collect thousands of examples, each a message with the ideal reply. The provider runs them through a copy of the model, adjusting its inner settings a little each time until its replies match. What comes back is a private model that behaves that way without being told.

  6. What training is good at

    Training earns its cost in three situations. A style or format too subtle to describe, which thousands of examples can show. A narrow job done so many times that a long prompt becomes expensive. And making a small, cheap model do one task as well as a large one. All three are about behaviour.

  7. What training is bad at

    Training is a poor way to teach facts. The model learns a blur of the examples, not a record of them. It cannot say where it learned something. It still invents answers when unsure. And the day after training ends, a price changes, and the model is wrong with no way to fix it except training again.

  8. Which one would you choose?

    Try the owner's request on a real case before going on.

  9. What training really costs

    The fee the provider charges is the smallest part. Examples have to be gathered, cleaned and checked by someone who knows what a good reply is. A test set is needed to prove the trained model is better. And when the provider retires the base model, all of it is done again. HM Nayem estimates six weeks before the first trained reply.

  10. A case where it pays

    The shop does have one job that fits. Every incoming message is sorted into one of forty kinds, a million times a month. A large model with a long prompt does it well, at a price. A small model trained on fifty thousand sorted messages does it as well, in a tenth of the time, for a twentieth of the cost.

  11. A large model teaches a small one

    Where did fifty thousand sorted messages come from? Nobody sorted them by hand. The large model did, over a weekend, and a person checked a sample. Its answers became the small model's training examples. Using a strong model to produce training material for a weaker, cheaper one is called distillation, and it is how most trained models get their examples today.

  12. The order to try things

    So there is a ladder, and teams climb it from the bottom. Each rung costs more than the one below and is harder to undo. Most products never need the top one. The ones that do, reach it knowing exactly which behaviour the lower rungs could not produce, which is also what tells them what examples to collect.

  13. The answer to the owner

    HM Nayem's reply is short. The assistant will know the shop, because it will read the shop's own pages every time it is asked. It will sound like the shop, because it is told how. One small model will be trained, for the sorting job only. The owner gets what she wanted. It just was not called training.

Tell it, then show it. Train it last.

Ideas in this story