What a trace should tell you about an agent run

There is a version of agent observability that is just logging with extra steps. You wrap each model call, you get a flat list of requests and responses, and when something goes wrong at three in the morning you scroll. It is better than nothing. It is not what we set out to build.

The question a trace should answer is not what did the agent do — you can usually guess that. It is why did this run diverge from the ones that worked. Those are different questions, and they need different span data.

Structure before content

The first thing we record is shape. A run that normally makes four tool calls and made eleven this time is interesting before you read a single prompt. A retrieval step that usually returns eight chunks and returned zero is interesting even if the model recovered gracefully — especially then, because a graceful recovery is a failure you will not notice until it compounds.

So every span carries the counts that describe its shape:

• Retrieval spans carry chunks.returned, chunks.requested, and the score of the lowest-ranked chunk that made the cut.
• Tool spans carry the argument names the model supplied, and whether validation passed.
• Model spans carry the stop reason. Not just stop versus length — whether a tool call was truncated mid-argument, which is a failure mode that reads as a parsing bug three layers away.

None of this requires capturing prompt content. Teams who cannot send us payloads still get the interesting half.

The decision points are the spans that matter

An agent run is a sequence of branch points. The model decided to call a tool rather than answer. The router decided this request was simple enough for the small model. The retry policy decided the timeout was transient.

Each of those is a decision made by code you wrote, on evidence you had at the time — and each is a place where a run goes wrong in a way that is invisible downstream. We give them their own span type. A decision span records the inputs the branch saw, the branch taken, and, where it is cheap to compute, the branches not taken.

That last part is the one people push back on, because it sounds expensive. It usually is not. A router that picked the small model can record the score that made it confident, which is a float you already had. When you are staring at a bad run, knowing the router scored it 0.31 against a 0.30 threshold is the entire investigation.

Timing is evidence, not decoration

The most common real bug we see is not an error. It is a step that succeeded in two milliseconds when it should have taken two hundred. Empty results are fast. Cache hits on a stale key are fast. A vector query against an index that finished rebuilding an hour ago is very fast.

Duration outliers on the low side are underused. Every alerting tool ships p99 latency breaches out of the box; almost none of them will tell you a span got suspiciously quick. We flag both tails, and the low tail catches more real bugs.

What this looks like in a span

Concretely, a retrieval span carries the numbers that let you rule it in or out without opening the payload:

with trace.span("retrieve.clauses", kind="retrieval") as span:
    hits = index.query(embedding, top_k=40, min_score=0.72)
    span.set("chunks.requested", 40)
    span.set("chunks.returned", len(hits))
    span.set("chunks.min_score", min(h.score for h in hits) if hits else None)

Three attributes, all of which you already computed. When chunks.returned is 0 and the span finished in three milliseconds, the trace has already told you what went wrong.

What we do not record

We do not record a “confidence score” for the run as a whole. We tried. It was a number people trusted more than it deserved, and it flattened exactly the structure we spend the rest of the trace preserving. If a single number could tell you an agent run was fine, you would not need a trace.

The trace is the artifact. The scoring belongs in evals, where it can be versioned, argued with, and re-run against history when you change your mind about what good means.