It forgot my name: AI Engineering, story 3
HM Nayem says hello, and one message later the assistant has forgotten the name. Find out where a chatbot's memory really lives.
The model remembers nothing; the app resends the whole conversation, and that has a size limit.
The story, step by step
HM Nayem says hello
HM Nayem says hello and gives a name. The assistant uses it straight away. So far, so good. Now watch what happens with the next message.
It forgot
One message later, it has no idea who it is talking to. This shop's first version sent the model only the newest message. That reveals something surprising about models.
Every request starts from nothing
A model keeps no memory at all. When a request ends, it is gone. The next request arrives at what is, in effect, a brand new model that has never met HM Nayem.
So the server keeps the record
The fix is not in the model. It is in the shop's code. The server saves every message, HM Nayem's and the assistant's, as the conversation goes.
And sends all of it, every time
With each new message, the server sends the whole conversation again from the top. The model reads it all fresh and answers as if it had been there the whole time.
Now it knows
It is a trick, and every chatbot uses it. The memory you feel in a conversation is your own words being read back to the model, over and over.
The pile grows every turn
Each turn adds two messages: what HM Nayem wrote and what the assistant replied. And every request carries all of them. The fiftieth request is far longer than the first, even when the new message is a single word.
There is a catch
Everything the model reads has to fit in its context window: a fixed amount of space, measured in tokens. The reply it writes needs room in there too. A short chat barely uses any of it.
Long chats fill it
But every turn is added and sent again, so the pile only grows. After a few hundred messages, or one pasted document, the window is nearly full. Each request also costs more and takes longer.
Past the limit, nothing works
If the server sends more than the window holds, the model does not quietly drop the oldest part. The whole request is refused with an error. A chat that worked for weeks simply stops.
The server tidies up
So the server chooses what to keep. Old turns are replaced by a short summary, and recent ones stay word for word. Deciding what the model sees is one of the main jobs in AI engineering.
Now at real speed
From the phone, none of this is visible. It just feels like talking to something that remembers. Behind every single message: fetch the conversation, trim it to fit, send it all, save the reply.
The memory is in the app, not the model.
- A model forgets each request the moment it ends.
- The app saves the conversation and resends all of it every time.
- Everything sent has to fit in the model's context window.
- When it gets full, the app summarises old turns and keeps recent ones.
Ideas in this story
- No memory: A model forgets every request the moment it ends. Your code resends the conversation each time.
- The context window: The fixed amount of text, counted in tokens, a model can take in at once.