What is a model's lack of memory?
It feels like a chatbot remembers what you said a minute ago. It does not. Each request to a model starts from nothing.
The memory is your code. It keeps the conversation and sends all of it again with every new message: the instructions, each earlier message, and then the new one.
This is why long conversations cost more and eventually run out of room, and why deciding what to resend is a real engineering job.
Why does a model's lack of memory matter?
People assume a chatbot remembers the conversation. The model does not. Every appearance of memory in an AI product is built by ordinary code that saves text and sends it again. Knowing this explains why conversations have a length limit, why long chats cost more, and where a product's real memory lives.
How does a model's lack of memory work?
- A person sends a message. Your server sends it to the model and gets a reply.
- The model keeps nothing. As far as it is concerned, that call never happened.
- Your server saves both the message and the reply.
- When the next message arrives, the server sends the whole saved conversation again, with the new message at the end.
- The model reads all of it afresh, and so appears to remember.
When does this matter?
It matters when
- A conversation has more than one turn
- An agent works through many steps
- A person returns another day and expects to be known
It does not matter when
- Each request stands alone, such as sorting one message
- You are processing documents one at a time
Where does a model's lack of memory sit in a real system?
The memory lives in your system, in a store keyed by conversation. Several copies of your server can then serve the same person, because none of them holds the conversation itself. Memory across days is a separate store of short facts.
What goes wrong with a model's lack of memory?
- Keeping the conversation only in one server's memory, so it vanishes when that server restarts.
- Sending the entire history for ever, until it no longer fits or costs too much.
- Believing the model learns from your conversations. A call does not change the model.
What does a model's lack of memory look like in practice?
A customer says "My name is Sam" and then asks "What is my name?". If the server sends only the second message, the model cannot know. If it sends both, the model answers "Sam". Nothing was remembered. The name was simply in the text it was given.