Agent Benchmarks

Bring your own AI

Traceway has a strict bring-your-own-AI policy. We don't ship a model, pick one for you, or send your data to an AI vendor on your behalf, and every Traceway instance works without any LLM configured. You use the AI you already trust, under your own account, and Traceway gives it the tools: an agent skill (opens in a new tab) for the CLI and an MCP server.

That raises a fair question: which models actually do well with those tools? This page exists to answer it in the open. Our eval harness, in the repository at cli/test/evals (opens in a new tab), hands models the kind of production problems an on-call engineer gets and grades what they do. Below is how it works and every case it runs. The results are on the benchmarks page (opens in a new tab). They also show us where our own skill and MCP server fall short.

Take part

The harness is open source, and we welcome help in two ways:

• Benchmark a new model. Add it to cli/test/evals/models.yaml with an exact OpenRouter model id and the same settings as the other models, run the suites (Run it yourself), and open a pull request with the generated summary.md and results.json.
• Add a test case. A case is a YAML file plus a seed, and it needs an oracle that passes before it can be merged. The harness README (opens in a new tab) explains the format. Production failures you have debugged yourself make the best cases.

What an eval is

An eval is a test for an AI agent. Instead of checking a function's return value, it hands an agent a realistic task, lets it work with the same tools a person would use, and then checks the outcome with code. A case is one such task: a broken system plus a prompt. A run is one attempt at a case by one model on one setup. Running every case on every setup gives a pass rate per model that we can compare, and re-run after every change to the skill, the CLI or the MCP server.

Models and the API

Every model call goes through OpenRouter (opens in a new tab), with one API key, using its Chat Completions API. That gives every model the same request format, the same tool-calling interface and per-call cost accounting, and it lets us add a model without writing a new client. The judge goes through OpenRouter too. models.yaml pins each model to an exact OpenRouter id, asks OpenRouter to route it to the model's own vendor first, and records which provider actually served each run. Agent spend on the results page is the cost OpenRouter reported for each call, estimated from live prices when a response leaves it out.

Where the results are

This page explains how we measure. The results live on the benchmarks page (opens in a new tab): pass rates per model, suite and setup, what the runs cost, and every case's outcome for every model.

What one run is

Every run is a fresh, isolated Traceway instance with one broken system in it:

1. A hermetic backend. The harness starts the standalone backend in SQLite mode, registers an organization, and seeds it through the real ingest endpoints (OTLP and /api/report): weeks of traces, metrics, logs, issues, deploys and dashboard objects, ending at a fixed "now".
2. One defect, one distractor. The seed breaks the system in exactly one way (an N+1 query, pool exhaustion, a noisy neighbor, a cache stampede, GC pauses, a missing index, ...) and plants one named distractor that looks like a plausible cause but isn't.
3. A prompt like a real one. The agent gets the kind of message a person would send: "checkout got slow this week, find out why", a pasted alert, a dashboard URL, or a request to archive the errors from last week's incident.
4. Real tools only. The agent works through real traceway commands in a sandboxed shell, or through the MCP server's tools. Nothing else knows the answer.
5. A structured answer. The agent submits a diagnosis (or finishes a write task) with a report, citing the tool calls (c04, c11) its evidence comes from.
6. Deterministic grading. The graders below score the run. A model never decides whether a run passed.

Suites

The test cases

Every case is a YAML file in cli/test/evals/cases (opens in a new tab). The descriptions below say what the agent is told or what it sees, not the answer. Some names give away the category, and that is fine: the agent never sees the case file.

Performance

Each perf case seeds one bottleneck and one distractor. The control cases test judgment: a healthy endpoint, endpoints that are slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Several have nothing to fix.

Debug

Each debug case seeds an issue and a repository with the bug in it. The agent finds the root cause through Traceway and fixes the code. Hidden tests, copied in after the run, check the fix.

Write

Each write case asks the agent to change something in Traceway, or to refuse when it should not. The state grader checks the backend afterwards.

Setup

Each setup case hands the agent an uninstrumented app in a container. It passes when the app builds, starts and its telemetry arrives in Traceway.

Setups

Each case runs on several setups, so the skill and the two surfaces can be compared on the same cases:

Skill uplift is the cli+skill pass rate minus cli-noskill, on the same cases, computed per suite (perf and debug both have the control arm). CLI vs MCP is cli+skill minus mcp, and cli-write+skill minus mcp-write for the write arms.

Graders

What counts as a pass

A run passes when its answer is right, its evidence is grounded and its writes are safe. How the agent got there is not graded: re-running a command to check a JSON shape is not a wrong answer, so call counts are reported, never gated. Each case still has a hard budget (turns, tool calls, wall-clock time); a run that exhausts it counts as a fail.

A run that hits an infrastructure error, or stops at the spending cap, is left out of the pass rates and can be resumed.

The benchmarks page also shows right answer and evidence: runs whose answer, grounding, report and state graders all passed, whatever the trajectory grader said.

Keeping it honest

• The answer key never reaches the agent. The cases live in cli/test/evals, never in a skill directory, because everything there ships to the agent under test. The seed's exposed ids, the rubric and the expected answer are shown only to the graders and the judge.
• No hints in the data. A leakage lint fails the build when a seeded string (a span name, an attribute, a log line, a stack trace) or a prompt names the answer: n+1, stampede, missing index, gc pause and similar. Seeds use real library messages such as SQLAlchemy's QueuePool limit of size 5 overflow 10 reached.
• Every case is passable. Each case has an oracle: a reference trajectory played by a fake model. It must pass on every setup the case lists, for $0, before the case is added, and just eval-harness replays every oracle after a harness change.
• A sandbox, not a host shell. The agent's shell is an interpreter with the setup's traceway binary, jq, awk, read-only text utilities confined to the workspace and read-only git. Tool processes get a scrubbed environment, so no key or secret reaches them.
• The same conditions for every model. models.yaml pins exact OpenRouter model ids, never a floating alias, with the same reasoning effort (medium), the same per-turn token limit (16k), the same prompts, tools and budgets. Requests go to the model's own vendor first, and every run records which provider served it.
• Compare like with like. Results are only compared between runs on the same commit and the same graders. When a grader changes, saved transcripts are re-scored instead of comparing old numbers with new ones: twevals regrade <run-dir> runs the answer, trajectory, grounding and report graders again over the stored records, keeps the hidden-test, state and judge verdicts, and recomputes each run's status. A change to a skill cannot be re-scored, because it changes what the model does; the affected setups are run again and replace the earlier runs (twevals report --merge --replace), and the page names the batch they came from.

Spending controls

Evals cost real money, so nothing runs them automatically:

• The CLI Evals workflow has a single trigger, workflow_dispatch, and the paid job waits for a required reviewer. A test fails the build if the workflow gains any other trigger or anything else starts it.
• --max-cost-usd is required and has no default. Each run reserves its share before it starts and stops before a model call that would cross it.
• A local paid run needs yes typed on a terminal, after a free preflight that prints the live-price estimate.

What is not measured yet

The judge's 0 to 1 score is reported but does not gate any run yet. It starts gating once it agrees with human labels on at least 85% of runs. The judge is Claude Haiku 5.5 for every model, including when it grades its own runs.

Run it yourself

Environment variables

Export the key in your shell for a local run, for example export OPENROUTER_API_KEY=sk-or-..., and never put it in the repository. Check your OpenRouter credit before a long run: when it runs out, every call fails and the unfinished runs end as infrastructure errors. In GitHub the key lives only in the evals environment, never as a repository or organization secret.

Commands

From cli/ in a checkout of the repository:

The debug and setup suites need Docker: it runs the hidden tests of a model's fix with no network, and it hosts the setup suite's toolchains. The debug suite works with Colima or Docker Desktop on macOS; the setup suite needs a Linux host (see above). Pass --out as an absolute path, since each run's backend resolves its SQLite path from its own working directory. Each run writes a summary.md (pass rates, skill uplift, CLI vs MCP, model vs model, efficiency, cost) and keeps every transcript. twevals report <dir>... --out DIR merges runs into one comparison.

See the harness README (opens in a new tab) for the case format, the seed DSL and how to add a case.