AI Engineering, story 3It forgot my name

HM Nayem says hello

1 of 12

It forgot my name: AI Engineering, story 3

HM Nayem says hello, and one message later the assistant has forgotten the name. Find out where a chatbot's memory really lives.

The model remembers nothing; the app resends the whole conversation, and that has a size limit.

The story, step by step

  1. HM Nayem says hello

    HM Nayem says hello and gives a name. The assistant uses it straight away. So far, so good. Now watch what happens with the next message.

  2. It forgot

    One message later, it has no idea who it is talking to. This shop's first version sent the model only the newest message. That reveals something surprising about models.

  3. Every request starts from nothing

    A model keeps no memory at all. When a request ends, it is gone. The next request arrives at what is, in effect, a brand new model that has never met HM Nayem.

  4. So the server keeps the record

    The fix is not in the model. It is in the shop's code. The server saves every message, HM Nayem's and the assistant's, as the conversation goes.

  5. And sends all of it, every time

    With each new message, the server sends the whole conversation again from the top. The model reads it all fresh and answers as if it had been there the whole time.

  6. Now it knows

    It is a trick, and every chatbot uses it. The memory you feel in a conversation is your own words being read back to the model, over and over.

  7. The pile grows every turn

    Each turn adds two messages: what HM Nayem wrote and what the assistant replied. And every request carries all of them. The fiftieth request is far longer than the first, even when the new message is a single word.

  8. There is a catch

    Everything the model reads has to fit in its context window: a fixed amount of space, measured in tokens. The reply it writes needs room in there too. A short chat barely uses any of it.

  9. Long chats fill it

    But every turn is added and sent again, so the pile only grows. After a few hundred messages, or one pasted document, the window is nearly full. Each request also costs more and takes longer.

  10. Past the limit, nothing works

    If the server sends more than the window holds, the model does not quietly drop the oldest part. The whole request is refused with an error. A chat that worked for weeks simply stops.

  11. The server tidies up

    So the server chooses what to keep. Old turns are replaced by a short summary, and recent ones stay word for word. Deciding what the model sees is one of the main jobs in AI engineering.

  12. Now at real speed

    From the phone, none of this is visible. It just feels like talking to something that remembers. Behind every single message: fetch the conversation, trim it to fit, send it all, save the reply.

The memory is in the app, not the model.

Ideas in this story