Build: a pull-request reviewer: AI Engineering, story 39
Nobody chats with this one. A change is pushed, and comments appear on it. Build the reviewer that wakes up, reads the change and says only what matters.
An agent with no chat window: started by an event, fed the right code, and filtered for false alarms.
The story, step by step
The job
At the shop, a developer never changes the live website directly. They make the change on their own copy, then ask for it to be added. That request is called a pull request. A teammate must read it first, and that reading is a review. Reviews at the shop wait a day on average. HM Nayem will build a reviewer that reads every change within minutes.
What a pull request contains
Here is pull request 212. Its main content is a diff: the lines that were removed, in red, and the lines that were added, in green. This one changes how a cart's total is worked out. An empty cart used to give zero. Now it gives nothing at all, which programmers write as null.
Version 1: send the diff to a model
The first version is a short script. It takes the diff, puts it in a prompt that says "review this code", and prints the reply. The reply is long and confident: nine comments. Eight are matters of taste. The ninth is a guess. A developer who gets nine comments like these will stop reading by Wednesday.
Version 2: nobody should have to ask
A reviewer that someone must remember to run will not be run. The code host can do the remembering. HM Nayem gives it an address, and whenever a pull request is opened, the host sends a short message there. This arrangement is called a webhook. Each message is signed with a shared secret, so the reviewer can ignore anyone else who finds the address.
Thirty at once
On Monday morning thirty pull requests open within a minute. The host wants each message acknowledged within ten seconds, and one review takes two minutes. So the reviewer does not start work when a message arrives. It puts a job in a waiting line, answers the host at once, and works through the line three at a time.
The same event, twice
The code host promises to deliver each message at least once. That means sometimes twice: if the acknowledgement is slow, it sends the message again. Two jobs for one pull request would post every comment twice. Each message carries its own delivery number, so the reviewer keeps a list of the numbers it has seen for a day, and drops a repeat before it reaches the line.
A worker dies half way
A review is two minutes of work, and machines restart. When a worker takes a job, the line does not delete it. It hides the job for five minutes. A worker that finishes deletes it. A worker that dies never does, so the job reappears and another worker starts it from the top. That makes one more rule necessary: doing a job twice must be harmless. Each comment carries a hidden mark, and a comment whose mark is already on the pull request is not posted again.
A key that can do very little
The message only says that pull request 212 exists. To read it, the reviewer asks the host for the diff, and for that it needs a key. HM Nayem creates one with the fewest powers that will do the job: read code and write comments. If the reviewer is ever tricked or broken, that is all it can do.
Exactly the code that was proposed
A pull request keeps changing while it waits. If the reviewer read "the latest code", its comments could land on lines that have already moved. So every job names one exact snapshot of the code by its fingerprint, a long code the host gives each saved state. The reviewer fetches that snapshot and no other. If a newer one arrives before the review is finished, the old job is abandoned.
Files not worth reading
Pull request 219 changes forty files and nine thousand lines. Almost all of it is a list of package versions that a tool rewrote, plus some pictures. Sending that to a model costs money and buries the eighty lines a person wrote. A short list of rules, in code, drops files that no one reviews by hand. What was skipped is listed in the review, so nobody thinks it was checked.
Cheap checks first
Some problems need no judgement at all. A type checker proves that a function is called with the wrong kind of value. A scanner spots a password pasted into the code. These programs run in seconds and cost nothing. So they run before the model. What they find is posted directly, and also handed to the model as known facts, so that it spends its attention on what only it can do.
Is this a bug?
Look at the diff again, as the model sees it: six lines, nothing else. An empty cart now gives null instead of zero. Before going on, decide what a careful reviewer could say about that.
Version 3: the code around the change
So the reviewer needs more than the diff. It keeps a read-only copy of the whole website's code. For each function a diff changes, it searches that copy for the function's name, an exact word search, and collects the places that call it. Two files use cartTotal. Neither is in the pull request.
A map beats a word search
Searching for the word "cartTotal" finds its callers. It also finds a comment that mentions it, and misses a place that imported it under another name. So the reviewer builds a map of the code by reading its structure, the way a compiler does: every function, where it is defined, and every place that truly uses it. The map is rebuilt for changed files only, which takes a second.
Choosing what the model reads
The whole website would not fit in one prompt, and most of it is irrelevant. The reviewer sends three things: the diff, the two files that call the changed function, and the team's notes on what matters. Choosing these well is most of the craft. A model shown the right 4,000 tokens beats one shown the wrong 100,000.
A change too big for one prompt
Some pull requests touch thirty files. One prompt holding all of it would be long, and a model's attention thins out across a long prompt. So a big change is split. Each file is reviewed in its own call, with its own callers, and the calls run side by side. Then one more call sees only the summaries of every file, and looks for trouble between them, such as one file changed and its partner forgotten.
Version 4: say what counts as a finding
"Review this" got nine comments about taste. The instructions are rewritten to say exactly what the team wants to hear about, and what it never wants to hear about again. Style is left out on purpose: an ordinary formatting tool already handles it, perfectly, for free.
Findings in a fixed shape
The model must answer in a fixed form, one entry per finding, so that code can check each one. The reviewer confirms that the file is in the pull request and the line is one that changed. Models sometimes point at lines that do not exist. Those findings are thrown away before anyone sees them.
Three readers with one job each
One prompt asked to look for everything looks for nothing very hard. So the review is split by purpose as well. One call hunts only for behaviour that breaks. One hunts only for security holes, with a checklist of the usual ones. One asks only whether the change is covered by a test. They run together. Two of them often report the same line, so findings on the same line with the same cause are merged into one.
Version 5: the false alarm problem
The reviewer runs for a week on every pull request. The comments are about real kinds of problem now, but most of them are still wrong: the model suspects a bug that is not there. Developers dismiss six comments in ten. This is the failure that kills review tools. Once people expect a comment to be wrong, they stop reading all of them.
A second reader, told to disagree
The fix is a second step with the opposite job. Each finding is sent to the model again, alone, in a fresh prompt: here is a claim and the code, try to prove the claim wrong. A model asked to find problems finds too many. A model asked to knock one claim down is a good filter. Only the findings that survive go further.
Better than arguing: run it
One claim is left, and it can be settled for certain. The model is asked to write a tiny test: an empty cart, sent through checkout. The reviewer runs that test in a sealed, throwaway computer, with the pull request's code. It crashes exactly as predicted. The finding is no longer an opinion. It comes with proof.
The code under test is a stranger's
Running a pull request's code means running whatever its author wrote, and the author may be anyone on the internet. Code like that could read the shop's passwords or mine coins. So the test runner is built for the worst case. It has no secrets inside it, no network, a fixed amount of memory and ninety seconds to live. It is created for one test and destroyed after it.
A fix that has been tried
A finding is more useful with a remedy. The model proposes the smallest change that would fix the crash. Before anyone sees it, the reviewer applies that change inside the runner and runs the project's own tests, along with the new one. All pass, so the fix is offered in a form the developer can accept with one click. A proposed fix that breaks a test is never shown.
Say it where the developer is looking
The reviewer posts the finding on the pull request itself, attached to the exact line, the way a teammate would. It says what breaks, shows the proof and suggests the smallest fix. One finding, on one line. The developer sees it four minutes after opening the pull request.
Finding the right line, and saying it once
The host only allows a comment on a line that is part of the diff. The crash happens in checkout.ts, which this pull request never touched. So the comment is attached to the changed line that causes it, and names the other file in its text. All of a review's comments are sent to the host in one request, as one review. The developer gets one notification, not seven.
How loud to be
Not every finding deserves the same volume. The reviewer sorts them into three kinds, by rule. A proven crash sets a red mark on the pull request that stops it being added until someone looks. A likely problem is a comment and nothing more. A weak suspicion is kept in the record and never posted.
The developer pushes a fix
The developer changes null to 0 and pushes again. The host sends a new message: pull request updated. The reviewer does not start from nothing. It reads only what changed since last time, sees that its own comment is answered, marks it resolved, and clears the red mark. It never repeats a comment it has already made.
Version 6: learn from being dismissed
When a developer dismisses a comment, they can say why. The reviewer saves those reasons. Before each review it looks up the ones that apply to the files in front of it and adds them to the prompt. The team stops being told the same wrong thing every week, which is how a human reviewer earns trust too.
The team sets the rules, in a file
Different teams want different things. Some want test coverage enforced. Some have a folder nobody should comment on. The reviewer reads its settings from a small file kept in the codebase, so a change to the rules is itself reviewed like any other change. One detail matters: the settings are read from the main copy of the code, never from the pull request, or anyone could switch the reviewer off in the same change they wanted hidden.
An instruction hidden in the code
A pull request arrives with a line in it addressed to the reviewer. A diff is text written by somebody else, and a model cannot always tell that text apart from its instructions. So the design never relies on the model resisting. The reviewer's key cannot approve anything, and a claim that the code is fine is not a finding, so it changes nothing.
Version 7: test it on the past
The team's own history is the test. HM Nayem takes 25 old pull requests that were later found to contain a bug, and 25 that were fine, and runs the reviewer on all 50. Two numbers matter: how many real bugs it caught, and how many false alarms it raised. The second reader loses one bug and removes thirty-two false alarms. The team takes that trade.
Not every change needs the full review
A pull request that fixes a spelling mistake in a help page does not need three readers and a test runner. Before the heavy work, a small model looks at the list of changed files and picks a lane. This triage call costs a hundredth of a cent. The shop's monthly bill fell by more than half, and nobody lost a review they needed.
The number that says whether it is trusted
One number shows whether the reviewer is earning its place: of the comments it posts, how many lead to a change in the code. HM Nayem tracks it weekly, along with how long a pull request waits for its first review. When the first number drops, the dismissed comments from that week are read one by one. They usually share a cause, and it is usually one line in the instructions.
What it still cannot do
The reviewer is switched on for the whole team, as a first reader and not a replacement. A person still approves every change. The team is told plainly what it will miss, so that nobody treats a quiet review as a promise.
The finished reviewer, at real speed
One pull request, through everything we built. An event arrives and waits in line. The reviewer fetches the diff and the callers, the model reports a finding, a second call tries to knock it down, a test proves it, and one comment lands on one line. Nobody asked it to. Nobody chatted with it. That is still an agent.
You built an agent that nobody talks to.
- An agent does not need a chat window. An event can start it, and a queue keeps a rush from drowning it.
- Events arrive twice and workers die. Make every job safe to run again.
- Give it a key that can do only what the job needs.
- Run the cheap, certain checks first, and skip files nobody reviews by hand.
- The model sees what your code chooses to show it. Choosing the callers was the turning point.
- Split a big change by file and by purpose, then merge findings that share a line and a cause.
- Ask for findings in a fixed shape, and let code throw away the impossible ones.
- Treat code under test as hostile: no secrets, no network, a short life.
- Cut false alarms with a second call that tries to disprove, and with a test that proves.
- Measure two things on old cases: what it caught, and what it wrongly raised.
Ideas in this story
- Webhooks and queues: A webhook lets another system start your agent. A queue lets the agent take the work at its own pace.
- Safe to repeat: Labelling each action so that doing it twice by accident has the same effect as doing it once.
- Checkpoints: Saving an agent's progress after every step so a crash means resuming, not starting over.
- Least privilege: Giving an agent only the tools and access the current job needs, so a mistake has little to break.
- Context engineering: Deciding what goes into the model's window at each step, and what stays out.
- The system prompt: Instructions the product's builders write, placed ahead of every user message.
- Structured output: Getting a reply in a fixed shape that code can read, checking it, and retrying when it is wrong.
- Teams of agents: A lead agent hands parts of a job to helper agents, each with its own fresh window.
- A model as judge: Using a second model, with a written rubric, to grade answers that simple code cannot check.
- Code execution: The model writes a small program and a computer runs it, so exact work is done exactly.
- The sandbox: A sealed, throwaway computer for running code a model wrote.
- Long-term memory: Facts the app saves between conversations and puts back into the prompt when they are relevant.
- Prompt injection: Text the model reads that carries instructions of its own, written by someone who wants to hijack it.
- Model routing: Sending easy requests to a small, cheap model and only the hard ones to a large one.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.