It remembers me: AI Engineering, story 14
HM Nayem comes back a week later and the assistant still knows the right boot size. The model remembers nothing. So who does?
How an assistant recalls last week when the model itself remembers nothing.
The story, step by step
Monday
HM Nayem is back at the shop's chat, looking for boots, and mentions two things in passing: a size, and a rule about materials. The assistant uses both and finds a good pair.
The chat ends
Remember how a chatbot seems to remember: the server resends the whole conversation each time. That only lasts as long as the conversation does. When HM Nayem closes the chat, there is nothing left to resend.
A week later
HM Nayem opens a new chat and asks a question that only makes sense if the shop remembers. It does not. Nothing broke. This is simply what every chatbot does unless someone builds more.
Decide what is worth keeping
So the shop builds memory. First question: remember what? Not the whole chat. Most of it is small talk. When a conversation ends, the server asks the model to pick out the few facts that will still matter next time.
Save it outside the model
Those facts are saved in an ordinary database the shop owns, filed under the customer. The model is not changed or retrained in any way. Nothing has been taught to it.
Next time, look first
Now replay the following week. A new chat begins. Before calling the model, the server looks up what it has saved about this customer.
Into the prompt
The saved facts go into the prompt, between the shop's instructions and the new message. To the model, it is just more text to read. It cannot tell these lines came from last week.
It remembers
And there it is. The model did not remember anything. The shop's code remembered, and told it. From the phone, the two are impossible to tell apart.
A year of memories
After a year there are hundreds of saved facts, and most have nothing to do with boots. Sending all of them would waste the window. So the server searches its memories by meaning, the way it searched help pages, and sends only the closest few.
Memories go out of date
Then one day the size changes. If the server only ever adds, it now holds two sizes and will keep getting it wrong. Memory has to be edited, not just piled up. A stale memory is worse than none.
Whose memory is it?
One last thing. This is a file of personal facts about a real person. A good product lets that person see what is kept and delete any of it, and never saves the sensitive things at all.
Now at real speed
A new message arrives, the server looks up what it knows, adds it to the prompt, and the model answers like an old friend. Then anything new worth keeping is saved for next time.
The model forgets. The app remembers for it.
- The resent conversation only lasts as long as that conversation.
- To remember longer, the app picks out a few facts and saves them itself.
- Next time, it fetches the relevant ones and puts them in the prompt.
- Memories must be updated when they change, and the person can see and delete them.
Ideas in this story
- Long-term memory: Facts the app saves between conversations and puts back into the prompt when they are relevant.
- No memory: A model forgets every request the moment it ends. Your code resends the conversation each time.