AI Engineering, story 24Guards at both doors

An assistant with no guards

1 of 15

Guards at both doors: AI Engineering, story 24

Abuse comes in and a made-up promise goes out. Watch the checks on each side of the model, what they catch, and what they cost.

Guardrails check messages on the way in and replies on the way out. They are code and small models around the main one, and they have limits.

The story, step by step

  1. An assistant with no guards

    The shop's assistant talks to anyone, and three kinds of message arrive that it was never meant for. One is abuse. One asks it to write a school essay. One tells it to forget its instructions. With nothing between the customer and the model, the model answers all three as best it can, and its best is sometimes awful.

  2. Two doors

    The fix is not a better model. It is two checkpoints, one on each side of it. Everything coming in is checked before the model reads it. Everything going out is checked before the customer does. These checkpoints are called guardrails. Neither one is the model, and that is the point.

  3. Way in, first check: plain rules

    The cheapest checks need no model at all. Is the message absurdly long? Has this person sent two hundred messages in the last hour? Does it match a known attack word for word? Code answers these in less than a thousandth of a second, and they stop a surprising share of trouble.

  4. Way in, second check: what kind of message?

    Rules cannot tell a rude customer from a frustrated one. So next, a small model is asked a single question: which of these five kinds is this message? It is not asked to be helpful or clever. A narrow question like that is hard to argue with, and costs a fiftieth of a full reply.

  5. What a stopped message gets

    A stopped message never reaches the main model. The customer gets a fixed line written by a person, the same every time. That matters. If the main model wrote the refusal, a clever message could still steer what it said. A fixed line cannot be steered. The stop is also recorded, for reasons that come later.

  6. The free rider

    One visitor found the first version and used it all month as a free writing assistant. It cost the shop four hundred dollars. Nobody was attacked and nothing leaked. The shop simply paid for someone else's homework. The subject check stops most of this, and a limit on how much any one person may use in a day stops the rest.

  7. Way out, first check: things that must never leave

    Now the other door. The model has written a reply, and before the customer sees it, code scans it for a short list of things that must never appear, whatever the reason. A card number. Another customer's email address. A link to anywhere but the shop. Patterns find these exactly, with no judgement involved.

  8. Second check: a promise nobody made

    Patterns cannot spot an invented fact. The reply says the parcel will arrive tomorrow, and no tool result or help page said so. The small model is given the reply and the material it was built from, and asked: is every claim here supported by that material? One is not. The reply is held.

  9. Third check: advice it should not give

    A customer back from a winter walk asks how to treat numb, white fingers. The model knows a plausible answer. A shop's chat is the wrong place for it. The way out has a short policy list of subjects where the only acceptable reply points to a professional. Health is on it. So are legal threats and anything about someone's safety.

  10. When a reply fails

    A held reply is not shown and not simply thrown away. The model is asked once more, and told exactly which check failed, so it can write a reply without the problem. Most second attempts pass. If the second also fails, the customer gets a fixed line and the chat goes to a person. The assistant never sends something it could not approve.

  11. Guards cost time

    Four checks could add a second to every reply. Two things keep the delay small. The checks on the way in run at the same moment as each other, and the main model starts before the slowest has finished, ready to be cancelled. On the way out, the reply is checked in pieces as it is written, and shown a sentence behind.

  12. Guards make mistakes too

    A customer asks whether a camping knife can go in hand luggage, and is stopped as a threat. A guard can be wrong in two directions: letting through something bad, or stopping something innocent. Tightening one loosens the other. HM Nayem counts both every week, on real stopped messages, and reads every wrongly stopped one.

  13. Is a sentence a guard?

    The assistant's instructions contain the line: never reveal another customer's data. Decide what that line is.

  14. What guards cannot do

    Guards stop what they were built to look for, and nothing else. They do not make a wrong answer right. They do not replace narrow permissions on tools, or a person's approval before money moves. They are one layer. A product that is safe has several, each assuming the others will sometimes fail.

  15. One message through both doors, at real speed

    An ordinary question, at full speed. Rules and the small model pass it on the way in. The main model answers. Patterns and the small model pass the reply on the way out. Four checks, a twentieth of a second, and the customer notices nothing. That is what a guard should feel like, nearly all of the time.

Guards stand outside the model, on both sides.

Ideas in this story