Crash at step 40: AI Engineering, story 15
An agent is emailing 200 club members when its server restarts. Watch it pick up exactly where it stopped, without emailing anyone twice.
How long jobs save their place, resume, and make repeated steps harmless.
The story, step by step
A long errand
HM Nayem runs a hiking club with 200 members, and renewals are due. An agent can do this, but it means hundreds of rounds of the loop and about an hour. HM Nayem hands the job over and closes the laptop.
Round after round
The job runs on a server, not on the laptop, so it carries on alone. For each member, the model writes the email and asks the harness to send it. Thirty-nine members in, all is well.
Then the server restarts
Servers restart. An update is installed, a machine is replaced, the power blinks. The program running the harness is simply gone. And so is everything it was holding in its head: the record, the place it had reached.
Starting over is not an option
The obvious fix is to run the job again from the top. But 39 people would get the same email twice, and 20 minutes of work would be paid for a second time. For a longer job, it might never finish at all.
Save after every step
The real fix is boring and powerful. After every single step, the harness writes its progress and the record to a database. This is called a checkpoint. The harness no longer keeps anything important only in its head.
The same crash, again
Run the same hour again, this time with checkpoints. The server restarts at the same moment. The program dies just as before. But this time, the important part is not inside the program.
Pick up where it stopped
A fresh copy of the harness starts up. Its first act is to read the last checkpoint. It gets back the record and the place it had reached, and carries on with member 40 as if nothing had happened.
The dangerous moment
One gap remains. What if the crash lands after the email to member 40 goes out, but before “40 done” is saved? On restart, the checkpoint says 39. The harness would send member 40 the email again.
Make a repeat harmless
So every send carries a unique label, like a stamp. The email service remembers the stamps it has seen. When the same one arrives a second time, it quietly does nothing. Now retrying is always safe.
HM Nayem checks in
Because progress lives in a database and not inside a running program, anyone can ask how the job is going at any time. HM Nayem opens the laptop on the trail and takes a look.
Done
An hour after it began, the last email goes out. Two hundred members, two hundred emails, not one of them doubled. The crash in the middle left no trace at all.
A long job must be able to stop and start.
- Long agent jobs run on a server, and servers restart.
- A checkpoint after every step means a crash resumes instead of starting over.
- A unique stamp on each action makes an accidental repeat harmless.
- Saved progress also lets anyone check on a job while it runs.
Ideas in this story
- Checkpoints: Saving an agent's progress after every step so a crash means resuming, not starting over.
- Safe to repeat: Labelling each action so that doing it twice by accident has the same effect as doing it once.