Datasets & Prove the Lift
Run any two AI models over a labeled dataset, score every answer with an LLM judge, and see the measured improvement — then share the result as a public benchmark page.
Datasets let you turn a set of real tasks into a repeatable, scored benchmark. Prove the lift runs two models over that dataset and shows you exactly how much better one is than the other, on your data — not a leaderboard, your workload.
The idea
- Build or import a dataset — a list of rows, each an input (task) with an optional expected answer. Import from CSV / JSONL / JSON, or turn a failing run into a row.
- Run — score one model over every row. A scorer grades each answer: deterministic
(
exact/contains/regex/ JSON-schema) or an LLM judge with a rubric (correctness, faithfulness, tool-choice). Every judge result comes with a reason. - Prove the lift (A/B) — run a base model and a refined model over the same rows and see the delta: the score lift, per-row improvements and regressions, latency, and estimated token savings. The number is honest — a regression shows as a negative lift, never a faked win.
The judge
Grading is a small, cheap task, so the judge defaults to the cheapest model you've already connected (zero setup). A local Ollama judge is a first-class option for $0, fully-private grading — offered, never forced. A cost estimate is shown before every run. Invoked never hosts a judge model; grading runs on your keys or your machine.
Share a result
Any lift result can be published as a public benchmark page — a link anyone can open in the
browser, like a PC benchmark score. It renders the animated comparison, the per-row breakdown, the
models, and the headline lift, at a clean SEO-friendly URL
(/lift/<base>-vs-<refined>-ai-model-evaluation-<id>). Results are public by nature and contain
model names, scores, and truncated task text — never your full outputs or credentials.
Every published result also appears in the benchmark portal at /lift — a public,
browsable index of all shared results, filterable by model. Each page shows only measured
facts from the run (the models, the provider each was called through, pass rates, latency, and
per-row deltas); there are no authored model descriptions to go stale or mislead.
Why it matters
This is the "prove it" primitive: instead of claiming a refined agent is better, you show the lift, judged automatically, on real data — and that proven number is what a versioned, hosted agent carries into production.