Evals are becoming your IP. So where's the chain of custody?

September 26, 2026 · the looptail team

An argument is circulating from enterprise-software leaders and the executives of the big data-labeling firms: you can't automate what you can't measure, evals gate AI adoption in the enterprise, and a company's evals will become its main proprietary IP. We think the argument is right — and unfinished. IP is not a vibe. IP has custody.

The half everyone agrees on

The reasoning goes: enterprises are stuck at coding agents because code is the one domain with built-in verification. Everything else — support, claims, underwriting, ops — is a non-deterministic process nobody has instrumented, so nobody can say whether the agent is working, improving, or quietly regressing. The companies that build internal eval environments get compounding agent performance; the ones that don't get demos. Whoever owns the evals owns the improvement rate. So far, so right.

Your evals encode what your business means by good — the refund window you actually honor, the tone your regulator expects, the edge cases your industry punishes. That is judgment competitors don't have. Calling it IP is not hype; it's accounting.

What makes something IP

Nobody calls a whiteboard sketch IP. Things earn that word when they areversioned, attributable, provable, and portable. Source code qualifies because git made custody automatic: every change has an author, a diff, a history that survives the person who wrote it. Most eval practice today fails all four tests — a rubric in a doc, scores in a notebook, a prompt tweak shipped on a hunch, the reasoning in someone's head. That isn't proprietary IP. That's tribal knowledge with a dashboard.

The test is concrete: could you show an acquirer, an auditor, or a court what your eval said six months ago — and what changed in production because of it?

The chain of custody, concretely

If evals are the asset, five links need to be on record — each one signed, each one append-only, each one referencing the last:

• The standard — the rubric, versioned like code, because "criterion 2 changed in v3" is a fact you will need.
• The judgment — every verdict tied to the rubric version and judge that produced it.
• The pattern — failures clustered into tracked issues, deterministically, so the count is arguable math rather than a hunch.
• The test — every proposed fix replayed against a regression set before it ships, pass or fail on record.
• The decision — who approved, on what replay evidence, and whether the gate was forced. Forever.

This is exactly what a signed, append-only trail records:evaluate, issue, improve, andapprove events, hash-chained so history can't be quietly rewritten. Evidence as a by-product of working, not a compliance project bolted on after.

The trap in "evals are IP"

Follow the logic one step further than the posts do. If evals are your main proprietary IP, then the platform holding your rubrics, scores, and history holds your IP. Most eval tooling is a hosted database with your judgment inside it. IP you cannot export is not yours.

That is why the substrate has to be open. Looptail's trail format is apublic, Apache-licensed draft spec: rubrics live in your repo, trails are local-first files you can verify with one command and no vendor — including without us. The hosted platform adds continuous judging, receipts, and review, but the custody layer belongs to you by construction. If we ever disappeared, your evidence wouldn't.

Where this goes

The enterprises that treat evals as IP will, within a budget cycle or two, ask their vendors the questions IP always raises: where is it, who touched it, can we take it with us, can we prove it in a dispute. The eval platforms built as dashboards will retrofit custody badly. Start with custody and the dashboard is the easy part — that's the bet we've made. If evals become the crown jewels, the winners will be the teams who can prove theirs.

FAQ