Sampling without losing the incident

At 60 billion spans a month, somebody is going to ask about sampling. Usually it is the person who owns the bill, and usually the answer they want is “sample at one percent”, and usually that answer is wrong in a way that only becomes obvious during an incident.

Here is the shape of the problem and what we actually run.

Head-based sampling is cheap and forgets the wrong things

The classic approach decides at the root span: roll a number, and if it lands under your rate, keep the whole trace. It is cheap, it is stateless, and the decision propagates cleanly through context so you never end up with half a trace.

It also means that when a customer reports a bad run at 14:32, the odds that you kept it are exactly your sample rate. At one percent, you are telling that customer you threw away their evidence. You will then turn sampling off for a week, which defeats the purpose, and turn it back on the moment the bill arrives, which resets the cycle.

Tail-based sampling remembers correctly and costs a lot

The obvious fix is to decide at the end: buffer the whole trace, look at whether it errored or ran long, then keep or drop. This is correct. It is also expensive in the boring operational sense — you need somewhere to hold every in-flight trace until it completes, which at our ingest rate is a substantial amount of memory doing nothing useful most of the time, plus a decision about what to do with traces that never complete.

Pure tail-based sampling moves the cost from storage to the ingest tier. Sometimes that is the right trade. It was not the right trade for us as a default.

What we run: deterministic head sampling with error escalation

Three rules, in order.

Rule one: the sampling decision is a hash, not a coin flip. We hash the trace ID and compare against the rate. Same trace ID, same decision, on every service, forever, with no coordination. This is what makes it safe to propagate — a downstream service that never talked to us reaches the same conclusion independently.

The useful side effect is reproducibility. If you turn the rate up from 1% to 5%, every trace that was being kept is still being kept. You are strictly adding, never reshuffling. Dashboards do not jump.

Rule two: errors escalate retroactively. Every span buffers locally for the duration of its own trace, not globally. If any span in a process errors, that process flushes its buffered spans for that trace regardless of the head decision, and stamps the trace as escalated. Other processes that already dropped their spans cannot retroactively produce them — so you get a partial trace, and we mark it as partial rather than pretending it is whole.

A partial trace centred on the error is worth considerably more than no trace, and being honest about which parts are missing is worth more than either.

Rule three: the low tail escalates too. Spans that complete suspiciously fast against their own historical distribution escalate the same way errors do. This is the rule that earns its keep. Empty retrievals, stale cache hits and silently-failing tool calls all present as unusually quick successes, and they are invisible to every sampling policy that only cares about errors and slowness.

The number

In production across our own workloads, a 2% base rate with escalation retains 94% of traces that a human would later want to look at, at 6.8% of the storage of keeping everything. The gap between 2% and 6.8% is the escalation, and it is the only part of this system anyone has ever thanked us for.

What we would do differently

We shipped the escalation buffer before we shipped a way to see how much it was holding, and spent a quarter guessing. If you build this, instrument the buffer before you turn it on. The tool that watches everything else should not be the one flying blind.