Cut agent-incident acknowledgement from 40 minutes to under four

Lumen Freight runs a quoting agent that reads carrier rate sheets, checks live capacity across a dozen partner APIs, and returns a bookable quote. When it is wrong, a customer is quoted a price that cannot be honoured, and somebody has to call them.

The engineering problem was not that it was wrong often. It was that when it was wrong, nobody could tell why quickly, and the two plausible causes needed completely different responses.

Two failures that look identical

A carrier API returning stale capacity and a model regression in the extraction step both surface as “the quote was wrong”. The first is somebody else’s outage and the response is to fail over to a secondary carrier feed. The second is ours and the response is to roll back.

“We had a Grafana board for API health, a board for model latency, a board for quote accuracy, and a Slack bot that posted the bad quotes,” says Caro Vidal, SRE Lead. “Four places, four different time axes, and you are correlating by eye at 2am. Median time to even acknowledge what kind of incident we were in was about forty minutes.”

OTLP, no vendor SDK

Lumen was already fully instrumented with OpenTelemetry in Go and had no appetite for a second tracer. They pointed an OTLP exporter at Cinder’s ingest endpoint and changed nothing else. Their existing HTTP client spans, which already covered every carrier API call, appeared as children of the agent run without any additional work.

That turned out to be the whole fix. The carrier call and the model call were finally on one timeline, in one trace, with the quote they produced.

“The correlation was never hard,” Vidal says. “It was just that nothing put those two spans in the same picture. Once they were siblings, you can see it in a second — either the capacity span came back stale, or it didn’t and the extraction went sideways.”

What changed operationally

Alerts now link directly to the failing trace rather than to a dashboard, so the first click is the evidence rather than a starting point. The three dashboards were retired because nobody opened them any more. Median acknowledgement dropped to under four minutes, and the number of incidents escalated to the wrong team went to approximately zero.

The rotation stayed the same size. That was the point.