AI Engineering glossary
61 ideas, each on its own page. Every page answers the same questions: what it is, why it matters, how it works, when to use it, and what goes wrong.
- The model callOne web request out, one reply back. Your own server always sits in the middle.
- TokensThe small pieces of text a model actually reads and writes.
- StreamingThe reply is sent piece by piece as it is written, instead of all at once at the end.
- The system promptInstructions the product's builders write, placed ahead of every user message.
- Examples in the promptA few worked question and answer pairs. The model copies their pattern.
- Thinking before answeringLetting the model work a problem through before it writes the reply.
- No memoryA model forgets every request the moment it ends. Your code resends the conversation each time.
- The context windowThe fixed amount of text, counted in tokens, a model can take in at once.
- Chance and temperatureThe model picks each token by weighted chance. Temperature sets how adventurous it is.
- Structured outputGetting a reply in a fixed shape that code can read, checking it, and retrying when it is wrong.
- Made-up answersA model writes what is likely, so a wrong answer sounds exactly as sure as a right one.
- GroundingPutting the real text in the prompt and telling the model to answer only from it.
- ChunkingCutting documents into small pieces so the right piece can be found and handed to the model.
- Reading scanned pages (OCR)Turning a picture of a page into text, so that it can be searched and handed to a model.
- EmbeddingsTurning text into a list of numbers so that similar meanings end up close together.
- Retrieval-augmented generationRetrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.
- Rewriting the questionTurning what the person typed into a complete question that a search can actually find.
- Hybrid searchSearching by meaning and by exact words at the same time, then merging the results.
- RerankingA second, more careful model re-sorts the search results so the best few come first.
- Tool callingThe model cannot do anything itself. It asks your code to, and your code answers.
- Model Context ProtocolA standard plug between assistants and tools, so each tool is connected once and works everywhere.
- Code executionThe model writes a small program and a computer runs it, so exact work is done exactly.
- The sandboxA sealed, throwaway computer for running code a model wrote.
- The agent loopA model using tools round after round, deciding each next step from what just happened, until the job is done.
- The harnessThe program around the model that runs the loop: it calls the model, runs the tools and keeps the record.
- PlanningThe agent writes itself a to-do list, works through it, and rewrites it when reality disagrees.
- Context engineeringDeciding what goes into the model's window at each step, and what stays out.
- CompactionReplacing a long record of work with a short summary so the agent can keep going.
- Long-term memoryFacts the app saves between conversations and puts back into the prompt when they are relevant.
- CheckpointsSaving an agent's progress after every step so a crash means resuming, not starting over.
- Safe to repeatLabelling each action so that doing it twice by accident has the same effect as doing it once.
- Teams of agentsA lead agent hands parts of a job to helper agents, each with its own fresh window.
- EvalsA fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- A model as judgeUsing a second model, with a written rubric, to grade answers that simple code cannot check.
- Prompt injectionText the model reads that carries instructions of its own, written by someone who wants to hijack it.
- Least privilegeGiving an agent only the tools and access the current job needs, so a mistake has little to break.
- Asking firstPausing before an action that sends something out or cannot be undone, and waiting for a person to say yes.
- Masking private dataSwapping sensitive details for placeholders before text reaches the model or the logs.
- Access controlDeciding in code, not in the prompt, whose data a request is allowed to touch.
- Images as inputA model that accepts pictures cuts each one into small squares and reads them much like tokens.
- Voice assistantsSpeech turned into text, answered by a model, and turned back into speech, fast enough to feel like talking.
- A time budgetA fixed amount of time a reply may take, shared out between every step on the way.
- Computer useA model operating a screen like a person: look at a screenshot, choose one click or keystroke, look again.
- Prompt cachingThe provider remembers the unchanging start of a prompt, so repeats of it cost about a tenth and arrive faster.
- Model routingSending easy requests to a small, cheap model and only the hard ones to a large one.
- Batch jobsSending work that nobody is waiting for all at once, for results within hours at about half the price.
- Rate limitsA cap the provider sets on how many requests or tokens you may send each minute.
- Retries and backoffTrying a failed request again after a wait that grows each time, with some randomness.
- FallbacksA second model or provider that takes over when the first one is unavailable.
- The model gatewayOne internal service that every model call passes through, handling keys, limits, retries and fallbacks in one place.
- Webhooks and queuesA webhook lets another system start your agent. A queue lets the agent take the work at its own pace.
- Tool designShaping a tool's name, description, inputs and results so that a model picks it at the right time and uses it correctly.
- GuardrailsChecks placed before and after a model call, in code or small models, that stop certain messages going in and certain replies coming out.
- Response cachingKeeping an answer that was already worked out, and giving it again when the same question, or one that means the same, comes back.
- Grading agentsTesting an agent on what it did: the state it left behind and the steps it took, over many runs in a practice world.
- Changing modelsMoving a product from one model version to another without changing how it behaves for its users.
- Staged rolloutReleasing a change to an AI product a step at a time, with numbers to watch at each step and a quick way back.
- Fine-tuningTraining a private copy of a model on your own examples, so that it behaves a certain way without being told each time.
- Questions into queriesLetting a model turn a question in plain words into a database query, which code then checks, runs and reports.
- TracingRecording every step of a request, with what went in, what came out and how long it took.
- MonitoringWatching speed, cost, errors and quality across all requests, with alarms when something shifts.