AI Engineering, story 27Clicking for you

A website with no door for software

1 of 13

Clicking for you: AI Engineering, story 27

A campsite booking site has buttons for people and nothing for software. So HM Nayem's assistant uses it the way a person would.

How a model operates a screen: look, decide one action, look again.

The story, step by step

  1. A website with no door for software

    HM Nayem wants a campsite booked for the hiking club. Every tool so far has been a function the assistant could call. This park's booking site has none. It has pages and buttons, made for people with a mouse.

  2. So it uses the site like a person

    The assistant opens a real web browser inside a sandbox and goes to the park's site. From here it will work the way you would: look at the screen, do one thing, look again.

  3. Step one: look

    The model cannot see a browser. So the harness takes a screenshot, a plain picture of the page, and sends that. From the receipt story you know what happens next: the picture is cut into squares and read like tokens.

  4. Step two: choose one action

    The model replies with exactly one action, in a fixed shape: what to do, and where on the picture, as a position across and down. Here: click the button that says “Book a site”.

  5. Step three: the harness does it

    The model only asked. The harness is what moves the pointer to that spot and presses the button, inside the sandboxed browser. It is tool calling again. The tool just happens to be a mouse.

  6. Then look again

    The page changes, and the model has no idea what it changed to. So the harness takes a fresh screenshot and sends it. Look, act, look. It is the agent loop from the coding story, with eyes and a mouse.

  7. Typing works the same way

    Filling in a form is just more actions, one per turn. Click the date box. Type “Saturday”. Click the site box. Type “14”. Each one is followed by a new screenshot to see what happened.

  8. It is slow

    Every single step costs a screenshot, which is about 1,500 tokens, and a full call to the model. This booking takes fourteen steps. A person does it in under a minute. A proper software connection would do it in a blink.

  9. And it is fragile

    Websites are untidy. A cookie pop-up slides over the button the model was about to press. The click lands on the wrong thing. The only way the assistant finds out is the next screenshot, where it sees the pop-up and deals with it.

  10. The screen can talk to the model

    Remember the poisoned web page. Here the model reads everything on the screen, including whatever a page was written to say. So the same defences apply: the browser is sealed in a sandbox, and the assistant holds as few powers as the job needs.

  11. Stop before paying

    The last button is “Confirm and pay”. Spending money cannot be undone, so the harness does not let the model press it. It pauses and shows HM Nayem exactly what is about to be bought.

  12. Booked

    HM Nayem says yes. The harness presses the button, takes one last screenshot to read the confirmation, and reports back. A website built only for people has been used by software.

  13. A last resort, and a useful one

    Operating a screen is slower, costlier and shakier than calling a function. So it is the last thing to reach for. But a great deal of the world's software has only a screen, and now an assistant can use that too.

It looks, takes one action, and looks again.

Ideas in this story