Trace Analyzer

What it checks, and how

A trace is a tree of timed spans. The analyzer rebuilds that tree from the parent ids, then runs five deterministic checks over it. There is no model in the loop: the same trace always produces the same findings, which is what makes them testable.

• Critical path. Walk backwards from the end of the root span; at each step the child that finished last is on the path, and any time no such child covers belongs to the parent. Children that overlap an already-chosen later child ran in parallel and are off the path. The attributed time sums to the root span's duration.
• Repeated calls (the N+1 shape). 4+ sibling spans with the same service and normalised operation, together covering at least 5% of the trace (or 10+ repeats regardless). If they overlap in time the page says so — concurrent fan-out is bounded in latency but multiplies load downstream.
• Serial reads. Two or more consecutive read-like remote calls under one parent (HTTP GET/HEAD, SELECT, or a name starting with get, fetch, lookup, load, list, query, find, search) that would save at least 5% of the trace if run concurrently. Writes are deliberately excluded: ordering is usually the point.
• Hot spot / unaccounted time. A span holding 20%+ of the critical path and at least 20 ms of it, so a trivially fast trace stays quiet. A leaf is a hot spot; a span with children is time its children do not explain: uninstrumented work, queueing, lock waits, or a GC pause.
• Errors. Spans with an error status, an error tag, or an HTTP 5xx. A failing span under a successful root is reported as a swallowed failure, not a failed request.

Getting a trace in

The input is OTLP/JSON^{⁠[OTLP Specification]}: trace and span ids are hex strings and 64-bit timestamps are decimal strings, which the parser handles exactly rather than through floating point. The OpenTelemetry Collector's file exporter writes newline-delimited OTLP JSON^{⁠[OpenTelemetry Collector file exporter]}, one export per line, and that works too. Jaeger's JSON trace export is also accepted.

When the analyzer is wrong

• It cannot see data dependencies. Two reads that look independent may feed each other. The serial-reads finding is therefore capped at medium severity and always phrased as “if independent”.
• Names stand in for intent. “Read-like” and “same operation” come from span names and attributes. Poor instrumentation produces both false positives and misses.
• Partial and sampled traces. A tail-sampled or truncated export can be missing the spans that explain a gap. The analyzer flags exports with missing parents as partial; it cannot tell you what is absent.
• Clock skew. Spans from different hosts are placed on one timeline by their own clocks. A child that appears to start before its parent is usually skew, not a bug.
• Asynchronous work. Fire-and-forget spans and message-queue links are treated as ordinary children; the analyzer does not follow span links across traces.
• Size. Inputs are capped at 20,000 spans and 8 MB so the tab stays responsive. Export one trace, not a day of them.

Why the critical path, and not the slowest span

The slowest span is often not the one worth fixing. In a fan-out, only the slowest branch matters; the rest finish while the request is still waiting. Latency across many parallel calls is dominated by the tail^{⁠[Dean & Barroso, 2013]}, which is why shortening off-path spans moves nothing and why the analyzer ranks by time on the path rather than by raw duration.

Further reading

• Observability: Metrics, Logs, and Traces — instrumenting services so traces like these exist in the first place.
• SRE Guide to SLOs, SLIs, and Error Budgets — deciding which latency is worth chasing.
• Postgres EXPLAIN Cheat Sheet — the next step when the hot spot is a database span.

About this tool

The Trace Analyzer is the first working piece of BackendBytes Diagnostics, an early-access backend diagnostics project. The analysis code is plain TypeScript with unit tests; the thresholds quoted on this page are imported from it. When the analyzer disagrees with what you know about your system, trust your system and tell us on X or at [email protected].