What is computer use?
Some systems have no connection for software, only pages and buttons made for people. Computer use lets an agent work with them anyway. The harness takes a screenshot, the model looks at it and replies with one action, such as a click at a position or some text to type, and the harness carries it out.
Then it takes another screenshot and the loop repeats. It is the agent loop with eyes and a mouse.
It is slow, because every step costs a screenshot and a model call, and fragile, because pages change and pop-ups appear. It also means the model reads everything on screen, including text written to mislead it. So it runs in a sandbox, with the smallest powers that will do, and stops to ask before anything that costs money.
Why does computer use matter?
Most software has no way in for other software. It has screens, buttons and forms built for people. An agent that can look at a screen and operate it the way a person does can work with almost anything, including old systems and websites that will never offer a proper connection.
How does computer use work?
- The harness takes a screenshot of a browser or desktop.
- The picture, and often a list of the page's controls, goes to a model that accepts images.
- The model replies with a single action: click here, type this, scroll.
- The harness performs the action and waits for the page to settle.
- It takes a new screenshot, and the loop repeats until the task is done.
When should you use computer use?
Use it when
- The system offers only a screen, with no connection for software
- The steps vary too much for a fixed script
- A person would otherwise click through it by hand
Think twice when
- A proper connection exists. It is faster, cheaper and far more reliable
- The site's terms forbid automated use
Where does computer use sit in a real system?
The browser runs inside a sandbox controlled by the harness. The model never touches it. The harness takes every picture and performs every action, which is where limits and approvals are applied.
What goes wrong with computer use?
- Taking the screenshot before the page has finished loading.
- Letting it click anything labelled pay, send or delete without a person's approval.
- Giving the model a password. A password manager should type it straight into the browser.
What does computer use look like in practice?
Asked to book a campsite, an agent takes 23 steps: it closes a cookie banner nobody told it about, corrects a date that landed in the wrong month, and stops at "Confirm and pay" for approval. It took 70 seconds and cost 9 cents. A person would take 45 seconds.