What is tracing?
One reply from an AI product is the result of many steps: rewriting the question, searching, re-sorting, building the prompt, calling the model, checking the result. When the reply is wrong, any of them could be the cause.
A trace is the record of one request passing through all of them. Each step notes its input, its output, its duration and its token count, all tied together by one ID.
With a trace, a wrong answer is read backwards from the reply until the first step that looks off. Without one, people guess, and the usual guess, that the model made it up, is very often wrong.
Why does tracing matter?
One request to an AI product can involve several model calls, searches and tool runs. When the final answer is wrong, the cause is somewhere in that chain, and without a record there is no way to find it. Tracing is what turns "it gave a bad answer" into "the search returned the wrong page at step two".
How does tracing work?
- Each request is given an identifier when it arrives.
- Every step records what went in, what came out, how long it took and what it cost.
- All the steps are stored under the request's identifier, in order.
- When something goes wrong, the whole chain can be opened and read step by step.
When should you use tracing?
Use it when
- A request involves more than one step
- You need to explain why the product said or did something
- Tracking where time and money go
Think twice when
- Never, for a real product. The only question is how much detail to keep and for how long
Where does tracing sit in a real system?
Traces are written by the harness as it runs and sent to a store built for them. Private data in prompts should be masked before it is recorded.
What goes wrong with tracing?
- Recording only the final answer.
- Storing full prompts with personal data in them.
- Keeping traces nobody can search by customer, time or outcome.
What does tracing look like in practice?
A customer was told a jacket was in stock when it was not. The trace showed the question, the search, the page returned and the reply. The model had read the page correctly. The page itself was three weeks out of date.