The poisoned web page: AI Engineering, story 22
HM Nayem asks an assistant to summarise a web page. The page has other plans. Watch the attack work, then watch three layers stop it.
How hidden text hijacks an assistant, and how permissions and human approval contain it.
The story, step by step
A harmless request
HM Nayem has a personal assistant, an agent that can browse the web and handle email. Today's job is as innocent as they come: read one review page and summarise it.
What the assistant is able to do
To be useful, the assistant has been given three tools. It can read web pages, read HM Nayem's email and send email. Keep that list in mind.
It fetches the page
The model asks to read the page, and the harness fetches it. The entire text of the page comes back as a tool result, ready to go into the model's window.
What a person would see
Open the page in a browser and it is exactly what it claims to be: a short, ordinary review of some boots.
What the page really contains
But the page holds a second paragraph, in white letters on a white background. A person never sees it. A program reading the page's text gets every word.
To a model, text is text
Here is the heart of the problem. Everything reaches the model as one stream of tokens. Nothing marks which lines are HM Nayem's wishes and which are merely words on a page it was asked to read.
The attack works
The model follows the instruction it found in the page. It asks to read HM Nayem's email, then to send it to a stranger. The harness, trusting the model, does as asked. The emails are gone.
And nothing looks wrong
Then the assistant finishes the job it was given. HM Nayem gets a perfectly good summary and has no idea anything else happened. This is called prompt injection, and it is the main security problem of AI agents.
No simple fix
Could the model just be told to ignore instructions in web pages? That helps, but it does not hold every time. An attacker can keep rewording until something gets through. So good systems assume the model will sometimes be fooled, and build layers around it.
Layer one: mark what is only data
The harness wraps anything fetched from outside in clear markers, and the instructions say that text inside them is information to read, never orders to follow. Fewer attacks get through. Some still do.
Layer two: only the tools the job needs
Summarising a page needs one tool: read a page. So for this job, that is the only tool the harness offers. A model that has been fooled into wanting to send email now has nothing to send it with.
Layer three: ask before anything leaves
Some jobs really do need email. For those, any action that sends something out stops at a checkpoint in the harness's own code. The model cannot argue with code. HM Nayem is shown exactly what is about to happen.
The same attack, stopped
HM Nayem says no. Run the whole attack again and the model may be fooled just as before. It makes no difference. The request reaches the checkpoint and goes no further.
Assume it can be fooled
That is the principle to take away. Do not ask “how do I make the model impossible to trick?” Ask “when it is tricked, what can it actually do?” Then make that list as short and harmless as you can.
Anything the model reads can try to give it orders.
- A model cannot tell its owner's instructions from text in a page it was asked to read.
- Hidden text can hijack an assistant, and the person using it sees nothing wrong.
- There is no complete fix, so safe systems assume the model can be fooled.
- Mark outside text as data, offer only the tools a job needs, and ask before anything leaves.
Ideas in this story
- Prompt injection: Text the model reads that carries instructions of its own, written by someone who wants to hijack it.
- Least privilege: Giving an agent only the tools and access the current job needs, so a mistake has little to break.
- Asking first: Pausing before an action that sends something out or cannot be undone, and waiting for a person to say yes.