Traces

A trace is the lifeline of one logical operation as it moves through your system: a request, a queued job, a scheduled command, an LLM call. Traceway models traces the same way OpenTelemetry does: a tree of spans identified by a shared trace id.

Spans are the primitive

Spans are the ground truth. Every other concept in Traceway (Endpoints, Tasks, AI Traces, Issues) is derived from spans on ingestion, then stored in a dedicated table so the dashboard can answer "what's slow?" or "what's expensive?" in milliseconds without scanning every span.

OTLP span batch
    │
    ▼
┌───────────────────────────┐
│ classify by kind + attrs  │     ← promotion rules
└───────────────────────────┘
    │
    ├──► endpoints      ─┐
    ├──► tasks          ─┤  ←─ materialized views,
    ├──► ai_traces      ─┤      indexed for fast aggregation
    ├──► exception_…    ─┘
    │
    └──► spans_v2 (all accepted spans, including promoted ones)

You can think of endpoints, tasks, and ai_traces as cached projections of the spans table. They exist for one reason: aggregations like "P95 of every endpoint over 7 days" or "total LLM cost grouped by trace_name" need to be cheap. Scanning the full spans table to compute those every dashboard load would not be cheap.

The trade-off this design makes:

• Top-level dashboards (Endpoints, Tasks, AI Traces) are fast: they read a small, pre-aggregated table.
• The detail/waterfall view falls back to spans for the full timeline of children under any entity.
• Promotion happens at write time, not at query time, so the rules are baked in once per span.

What gets promoted

A span lands in a top-level table when its kind or attributes match a rule. The classifier doesn't care whether the span is a root or a child. It cares about what the span represents.

In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table. Endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.

Root and non-root entities

Each Endpoint / Task / AI Trace row carries an is_root flag.

• Root: the span had no parent in any process. This is the typical case: an inbound HTTP request, a standalone cron run, a fresh LLM call.
• Non-root: the span was triggered by another trace. Examples:
  • A queue worker's CONSUMER span parented to the dispatcher's PRODUCER span via the trace context serialized into the job payload.
  • A child gen_ai span made inside a request handler.
  • A cross-service inbound HTTP hop where the upstream service propagated traceparent.

Non-root rows surface a Non-root chip in the dashboard list. On the row's detail page, a View distributed trace link jumps to the full picture across all the entities sharing that trace id.

Every OTLP projection keeps the ids of the span it was made from: trace_id, span_id and parent_span_id. Its own row id is derived from the project, the trace and the span, so a retried export produces the same id. Native /api/report occurrences keep their SDK UUIDs instead; the API exposes UUID-width span IDs and OTLP export maps them to eight bytes.

Distributed traces

Every Endpoint / Task / AI Trace stores the trace_id of its span. Rows that share a trace id belong together, whichever service or project reported them. So:

• A request and its queue worker share a trace id when the instrumentation propagates a parent context. Some queue integrations start a new trace with a link instead; those runs are not grouped by shared trace ID. With parent-context propagation the card shows the endpoint and task together, nested by their span parentage. A task started as a new trace with a span link has its own trace view.
• A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same trace id. Same card.
• Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row, nested by the spans in between.

Standard W3C propagation also makes browser requests share the backend trace ID. OTel span links can reference another trace, but those traces remain separate. See Distributed Tracing.

What each entity stores

The shape mirrors what's useful for the dashboard's aggregations on that entity.

Endpoints

Identified by HTTP method + route template (GET /api/users/:id). Ranked by Impact, a composite of Apdex, error rate, P99, client-error rate, and volume-weighted error rate. Streaming responses (SSE / WebSocket) are flagged and excluded from latency-based components.

Unmatched requests (404s)

Requests that return 404 without matching any route are grouped under a single UNMATCHED endpoint. This keeps scanner and bot traffic from creating one endpoint row per probed path, and UNMATCHED is excluded from client-error impact scoring.

On the OTel ingestion path, a 404 from a matched route keeps its route identity. If your GET /users/:id handler deliberately returns 404 for a missing user, it stays grouped under GET /users/:id. A request only becomes UNMATCHED when:

• the span carries no valid http.route attribute, meaning the framework matched no route, or
• http.route is a catch-all made only of slashes and wildcards, such as /, /*, /**, or */*. Some instrumentations report these for unmatched requests: Express reports / when only app-level middleware ran, and Spring reports /** for its static resource handler.

Scoped wildcards like GET /api/* are real routes and keep their name even on 404. The native /api/report path retains its existing behavior of grouping all 404 responses as UNMATCHED; it cannot use the same OTLP route-attribute distinction.

Tasks

Identified by a task name (job class, scheduled command, agent operation). Ranked by count × (P95 − P50) as a rough impact proxy. CONSUMER spans from any queue driver (database, Redis, SQS, Beanstalk, …) land here, as do root console.command spans.

AI Traces

Identified by trace.name (your agent or workflow). Carries model, provider, input/output/cached/reasoning tokens, input/output/total cost, finish reason, and a pointer to the full conversation in object storage. Ranked by total cost by default.

Lifecycle and sampling

1. A span is created in your app (by a framework's auto-instrumentation or by manual OTel calls).
2. Child spans, attributes, and events accumulate against it.
3. The trace is exported via OTLP.
4. Traceway stores accepted spans in spans_v2, then writes matching entity projections. Parent IDs remain unchanged. Sampling, healthcheck filtering, permissions and rejected rows can limit what is stored.
5. Exceptions recorded as "exception" events become Issues attributed to the owning entity.

Sampling:

• Sample rate: percentage of normal traces recorded (default: 100%).
• Error sample rate: percentage of error traces recorded (default: 100%). Keep this at 100% to catch every failure.