Clicking for you: AI Engineering, story 27
A campsite booking site has buttons for people and nothing for software. So HM Nayem's assistant uses it the way a person would.
How a model operates a screen: look, decide one action, look again.
The story, step by step
A website with no door for software
HM Nayem wants a campsite booked for the hiking club. Every tool so far has been a function the assistant could call. This park's booking site has none. It has pages and buttons, made for people with a mouse.
So it uses the site like a person
The assistant opens a real web browser inside a sandbox and goes to the park's site. From here it will work the way you would: look at the screen, do one thing, look again.
Step one: look
The model cannot see a browser. So the harness takes a screenshot, a plain picture of the page, and sends that. From the receipt story you know what happens next: the picture is cut into squares and read like tokens.
Step two: choose one action
The model replies with exactly one action, in a fixed shape: what to do, and where on the picture, as a position across and down. Here: click the button that says “Book a site”.
Step three: the harness does it
The model only asked. The harness is what moves the pointer to that spot and presses the button, inside the sandboxed browser. It is tool calling again. The tool just happens to be a mouse.
Then look again
The page changes, and the model has no idea what it changed to. So the harness takes a fresh screenshot and sends it. Look, act, look. It is the agent loop from the coding story, with eyes and a mouse.
Typing works the same way
Filling in a form is just more actions, one per turn. Click the date box. Type “Saturday”. Click the site box. Type “14”. Each one is followed by a new screenshot to see what happened.
It is slow
Every single step costs a screenshot, which is about 1,500 tokens, and a full call to the model. This booking takes fourteen steps. A person does it in under a minute. A proper software connection would do it in a blink.
And it is fragile
Websites are untidy. A cookie pop-up slides over the button the model was about to press. The click lands on the wrong thing. The only way the assistant finds out is the next screenshot, where it sees the pop-up and deals with it.
The screen can talk to the model
Remember the poisoned web page. Here the model reads everything on the screen, including whatever a page was written to say. So the same defences apply: the browser is sealed in a sandbox, and the assistant holds as few powers as the job needs.
Stop before paying
The last button is “Confirm and pay”. Spending money cannot be undone, so the harness does not let the model press it. It pauses and shows HM Nayem exactly what is about to be bought.
Booked
HM Nayem says yes. The harness presses the button, takes one last screenshot to read the confirmation, and reports back. A website built only for people has been used by software.
A last resort, and a useful one
Operating a screen is slower, costlier and shakier than calling a function. So it is the last thing to reach for. But a great deal of the world's software has only a screen, and now an assistant can use that too.
It looks, takes one action, and looks again.
- When a system has no connection for software, an agent can use its screen.
- Each turn is a screenshot in and one click or keystroke out.
- The harness performs every action; the model only chooses it.
- It is slow and fragile, so it is the last resort.
- It runs in a sandbox and stops to ask before spending money.
Ideas in this story
- Computer use: A model operating a screen like a person: look at a screenshot, choose one click or keystroke, look again.
- Images as input: A model that accepts pictures cuts each one into small squares and reads them much like tokens.
- Asking first: Pausing before an action that sends something out or cannot be undone, and waiting for a person to say yes.