conediff

Diff two recorded AI runs across a model or tool swap. See exactly what changed in outcome, cost, and latency — with a per-step timeline.

What it does

You swapped a model, re-ordered a tool chain, or changed a prompt, and you have two recorded runs. conediff compares them deterministically and tells you what actually changed: the outcome diff, the cost delta, the latency delta, and a per-step attribution so you can see which step moved.

It is honest by construction. A field that is missing, untyped, or corrupt stays unknown — it is never silently coerced to zero, pass, or fail. The report is both machine-readable JSON and a human-readable timeline, and it is pure: the same input always produces byte-identical output.

Try the local CLI

No account, no upload, no provider keys. It runs on local fixtures:

git clone https://github.com/LibertyDigitalSystems/conediff.dev
cd conediff.dev
node src/cli.mjs fixtures/positive-outcome-cost-changed.json --text

Point it at a Langfuse-style trace pair to normalize a real recorded-run export into the comparison contract:

node src/cli.mjs --langfuse baseline-trace.json candidate-trace.json --pretty

Why it is trustworthy

• Deterministic and reproducible — no wall-clock, no randomness.
• Unknown is a first-class result, distinct from pass and fail.
• Cost is never understated; corrupt numbers are dropped, not guessed.
• Local-first: the free CLI needs no network and no hosted account.
• Versioned JSON contract (v1) for both input and output.

Get started

The CLI and fixtures are the free, local trial. Hosted run history and team CI policies are a proposed paid offer — not yet enabled.

Read the docs on GitHub