Train it, or tell it?: AI Engineering, story 8
The shop's owner wants a model trained on the shop's own data. See the three ways to change what a model does, and why training is usually the last one to reach for.
A model's behaviour is changed by telling, showing or training. Training suits style and narrow jobs at scale, and is wrong for facts that change.
The story, step by step
"Train it on our data"
The shop's owner has read that companies train their own AI. She asks HM Nayem to train one on everything the shop has. It sounds like the obvious way to make an assistant that knows the shop. It is one of three ways to change what a model does, and usually the last one worth trying.
Three ways to change a model's behaviour
You can tell it: put instructions in the prompt. You can show it: fetch the right text and place it in the prompt at the moment of asking. Or you can train it: change the model itself. The first two leave the model untouched and change what it reads. Only the third changes what it is.
Telling
The owner wants replies to be warm and short. HM Nayem adds two sentences to the instructions and three example replies. It takes ten minutes, and the next reply is warm and short. If it is wrong, it is changed back in ten more. For how a model should behave, telling is nearly always enough.
Showing
The owner also wants the assistant to know the returns policy, the prices and the opening hours. Those are facts, and facts change. So they are not put into the model at all. They are fetched from the help pages each time someone asks, as earlier stories showed. When a price changes, the page changes, and the very next answer is right.
Training
Training is different in kind. HM Nayem would collect thousands of examples, each a message with the ideal reply. The provider runs them through a copy of the model, adjusting its inner settings a little each time until its replies match. What comes back is a private model that behaves that way without being told.
What training is good at
Training earns its cost in three situations. A style or format too subtle to describe, which thousands of examples can show. A narrow job done so many times that a long prompt becomes expensive. And making a small, cheap model do one task as well as a large one. All three are about behaviour.
What training is bad at
Training is a poor way to teach facts. The model learns a blur of the examples, not a record of them. It cannot say where it learned something. It still invents answers when unsure. And the day after training ends, a price changes, and the model is wrong with no way to fix it except training again.
Which one would you choose?
Try the owner's request on a real case before going on.
What training really costs
The fee the provider charges is the smallest part. Examples have to be gathered, cleaned and checked by someone who knows what a good reply is. A test set is needed to prove the trained model is better. And when the provider retires the base model, all of it is done again. HM Nayem estimates six weeks before the first trained reply.
A case where it pays
The shop does have one job that fits. Every incoming message is sorted into one of forty kinds, a million times a month. A large model with a long prompt does it well, at a price. A small model trained on fifty thousand sorted messages does it as well, in a tenth of the time, for a twentieth of the cost.
A large model teaches a small one
Where did fifty thousand sorted messages come from? Nobody sorted them by hand. The large model did, over a weekend, and a person checked a sample. Its answers became the small model's training examples. Using a strong model to produce training material for a weaker, cheaper one is called distillation, and it is how most trained models get their examples today.
The order to try things
So there is a ladder, and teams climb it from the bottom. Each rung costs more than the one below and is harder to undo. Most products never need the top one. The ones that do, reach it knowing exactly which behaviour the lower rungs could not produce, which is also what tells them what examples to collect.
The answer to the owner
HM Nayem's reply is short. The assistant will know the shop, because it will read the shop's own pages every time it is asked. It will sound like the shop, because it is told how. One small model will be trained, for the sorting job only. The owner gets what she wanted. It just was not called training.
Tell it, then show it. Train it last.
- A model's behaviour is changed by telling, by showing, or by training.
- Facts belong in pages that are fetched, never in the model.
- Training suits style, format and narrow jobs done at huge scale.
- A trained model still makes things up and cannot name its source.
- The real cost of training is the examples, the testing and doing it again.
- A large model can produce the examples that train a small one.
Ideas in this story
- The system prompt: Instructions the product's builders write, placed ahead of every user message.
- Examples in the prompt: A few worked question and answer pairs. The model copies their pattern.
- Retrieval-augmented generation: Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.
- Fine-tuning: Training a private copy of a model on your own examples, so that it behaves a certain way without being told each time.
- Made-up answers: A model writes what is likely, so a wrong answer sounds exactly as sure as a right one.
- Model routing: Sending easy requests to a small, cheap model and only the hard ones to a large one.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.