AI Engineering, story 12Fix the failing test

One sentence, a whole job

1 of 14

Fix the failing test: AI Engineering, story 12

HM Nayem tells a coding agent one sentence: the checkout test is failing, fix it. Watch it look, read, change, check and try again, on its own.

An agent is a model using tools in a loop with a plan it keeps rewriting; the harness is the program running that loop.

The story, step by step

  1. One sentence, a whole job

    HM Nayem is building a website for a hiking club, and one of its tests has gone red. This time the message is not a question. It is a job. Nobody knows yet how many steps it will take, not even the agent.

  2. A new kind of program

    In earlier stories the shop's server called the model once or twice and stopped. This program is different. It keeps going back to the model, round after round, until the job is done. It is called a harness. A model running inside one is called an agent.

  3. What the agent can do

    Like the order lookup, each tool is an ordinary function with a short description. Three are enough for this job: read a file, change a file, and run the tests.

  4. First, a plan

    Before touching anything, the model writes itself a short to-do list. Nobody told it how to solve this. The list is its own guess at the steps, and it keeps a long job on track.

  5. Round 1: look

    Round one. The model asks for the tests to be run. The harness runs them and sends the result back. Now the agent knows what is failing: a total that should be 108 came out as 100.

  6. Round 2: read

    Round two. Nobody told the agent where the bug is. It follows the evidence: a wrong total, so read the code that works out totals. And there it is. A tax rate is passed in, and never used.

  7. Round 3: act

    Round three is the first one that changes anything. The model asks for one line to be edited so that tax is added. The harness makes the change in the real file.

  8. Round 4: check, and a surprise

    The agent does not assume its fix worked. It runs the tests again. They still fail, by a hair: a rounding error far past the decimal point. No plan written in advance could have predicted this.

  9. The plan changes

    So the agent rewrites its own list. This is the difference between an agent and a script. A script follows fixed steps. An agent chooses the next step from what just happened.

  10. Rounds 5 and 6

    One more edit, to round the total to two decimal places. One more test run. This time every test passes.

  11. Knowing it is done

    How does the loop end? Each round, the model either asks for a tool or it does not. This time it replies with plain words and no tool request. The harness takes that as “finished” and stops calling it.

  12. The report

    HM Nayem typed one sentence and gets one back. In between, the agent took six steps that nobody listed for it, including one it had to invent halfway through.

  13. What held it together

    Remember, a model has no memory. So how did round six know about round one? The harness kept the full record and sent all of it again each round. That growing record is the agent's whole working memory. It is also the subject of the next story.

  14. Now at real speed

    Look, read, change, check, change, check, done. An agent is nothing more than a model, some tools and a loop. What makes it feel clever is that each step is chosen from the result of the last.

An agent is a model, tools and a loop.

Ideas in this story