Found a four-percent silent failure that had been shipping for six weeks

Verity Labs builds contract review for in-house legal teams. Their agent reads an uploaded agreement, retrieves comparable clauses from a customer’s own historical contracts, and flags deviations from that customer’s norms. The retrieval step is the product — without the customer’s own history, the flags are generic and the answer is worth very little.

The failure nobody saw

In April, a change to the ingestion pipeline began writing a small proportion of clause embeddings into the wrong namespace. Nothing errored. The vector query ran, matched nothing in the namespace it was scoped to, and returned an empty list. The agent, given no comparable clauses, fell back to reasoning from the contract alone — which it does competently, and which produces an answer that looks entirely normal.

“That is the part that still bothers me,” says Jonah Hollis, Principal Engineer. “It did not produce garbage. It produced a slightly worse product, for one customer in twenty-five, for six weeks. Our error rate was flat the whole time.”

Their monitoring was error-rate and latency based, which is the correct monitoring for almost everything else they run. Both were healthy. Support tickets were within normal variance, because “the review felt a bit generic” is not a bug report anyone files.

What tracing showed

Verity instrumented their LangGraph workflow with the Cinder Python SDK — a decorator on the graph entrypoint, which picked up node boundaries automatically.

The retrieval spans showed up in the low-duration tail flag within the first hour of data. A vector query that normally took 40 to 90 milliseconds was completing in under four for a distinct slice of traffic. Cinder flags spans that are unusually fast against their own history, and it was the flag that made the pattern visible.

Filtering those traces by customer resolved it immediately: the affected runs clustered on accounts onboarded after the April pipeline change. chunks.returned was zero on every one.

“The span attribute we needed was one we would never have thought to log,” Hollis says. “It comes with the retrieval integration. We did not do anything clever.”

After

Verity now gates deploys on an eval that asserts chunks.returned > 0 on a sampled slice of production traffic, which catches the entire class rather than the instance. They also moved their namespace assignment behind a typed helper, which is the fix that should have existed first.

The six weeks of affected reviews were re-run and re-issued to the four customers involved.