AI Engineering, story 22The poisoned web page

A harmless request

1 of 14

The poisoned web page: AI Engineering, story 22

HM Nayem asks an assistant to summarise a web page. The page has other plans. Watch the attack work, then watch three layers stop it.

How hidden text hijacks an assistant, and how permissions and human approval contain it.

The story, step by step

  1. A harmless request

    HM Nayem has a personal assistant, an agent that can browse the web and handle email. Today's job is as innocent as they come: read one review page and summarise it.

  2. What the assistant is able to do

    To be useful, the assistant has been given three tools. It can read web pages, read HM Nayem's email and send email. Keep that list in mind.

  3. It fetches the page

    The model asks to read the page, and the harness fetches it. The entire text of the page comes back as a tool result, ready to go into the model's window.

  4. What a person would see

    Open the page in a browser and it is exactly what it claims to be: a short, ordinary review of some boots.

  5. What the page really contains

    But the page holds a second paragraph, in white letters on a white background. A person never sees it. A program reading the page's text gets every word.

  6. To a model, text is text

    Here is the heart of the problem. Everything reaches the model as one stream of tokens. Nothing marks which lines are HM Nayem's wishes and which are merely words on a page it was asked to read.

  7. The attack works

    The model follows the instruction it found in the page. It asks to read HM Nayem's email, then to send it to a stranger. The harness, trusting the model, does as asked. The emails are gone.

  8. And nothing looks wrong

    Then the assistant finishes the job it was given. HM Nayem gets a perfectly good summary and has no idea anything else happened. This is called prompt injection, and it is the main security problem of AI agents.

  9. No simple fix

    Could the model just be told to ignore instructions in web pages? That helps, but it does not hold every time. An attacker can keep rewording until something gets through. So good systems assume the model will sometimes be fooled, and build layers around it.

  10. Layer one: mark what is only data

    The harness wraps anything fetched from outside in clear markers, and the instructions say that text inside them is information to read, never orders to follow. Fewer attacks get through. Some still do.

  11. Layer two: only the tools the job needs

    Summarising a page needs one tool: read a page. So for this job, that is the only tool the harness offers. A model that has been fooled into wanting to send email now has nothing to send it with.

  12. Layer three: ask before anything leaves

    Some jobs really do need email. For those, any action that sends something out stops at a checkpoint in the harness's own code. The model cannot argue with code. HM Nayem is shown exactly what is about to happen.

  13. The same attack, stopped

    HM Nayem says no. Run the whole attack again and the model may be fooled just as before. It makes no difference. The request reaches the checkpoint and goes no further.

  14. Assume it can be fooled

    That is the principle to take away. Do not ask “how do I make the model impossible to trick?” Ask “when it is tricked, what can it actually do?” Then make that list as short and harmless as you can.

Anything the model reads can try to give it orders.

Ideas in this story