Eval-driven development: a practical loop for teams shipping weekly
July 12, 2026 · the looptail team
Teams shipping AI weekly have quietly re-invented a discipline software already knows: don't ship what you can't test. The AI version isn't unit tests — it's evals you wrote, run continuously, gating changes. Here's the practical loop, sized for a team that ships every week.
The weekly cadence
- Monday — propose. Changes come from evidence: last week's failure clusters, outcome signals, incidents. Each proposed change names the failures it should fix.
- Tuesday — replay. Score the candidate against the regression set (recorded real traffic, not synthetic fixtures) with your rubrics. The gate is your own pass threshold, versioned in the rubric file.
- Wednesday — canary and ship. A slice of live traffic, rubrics still watching. Promote when the scores hold; the change lands with its evidence attached.
- All week — observe. Every decision recorded; failures clustered into next Monday's proposals. The set never rots because production keeps feeding it.
Write rubrics like you mean them
The failure mode of eval-driven development is generic metrics: helpfulness scores that never catch a policy violation. A rubric earns its place when each criterion is checkable from recorded evidence and traceable to something that actually went wrong. "Quoted refund window matches the published policy" caught a real bug; "response is helpful" never will. Version rubrics like code — a bumped version re-scores history, which is how you audit a standard's evolution. (This is what looptail-evals runs, with judge models on your keys.)
The part that compounds
Gates make weeks safe; records make quarters valuable. If every pass through the loop — decision, verdict, change, approval, outcome — lands in an append-only,signed trail, you accumulate two assets: a regression set that mirrors reality, and evidence that your improvement process is controlled (which is precisely whatthe EU AI Act's change-control expectations ask for). Experimentation IDEs likeBraintrust are where hypotheses get explored; the loop is how the winners ship without breaking last month's fixes.
FAQ
What is eval-driven development?
A working method for AI systems where changes are gated on evaluations the team wrote themselves: encode your standards as rubrics, score candidate changes against recorded real traffic before shipping, and let production outcomes feed new cases back into the eval set. It is test-driven development adapted to systems whose behavior is probabilistic and whose inputs drift.
How is a rubric different from a unit test?
A unit test asserts an exact output; a rubric scores a judgment call — “quotes the correct refund window”, “escalates disputed denials”. Rubrics are versioned like code and applied by an LLM judge, which makes them robust to paraphrase and applicable to live traffic, not just fixtures.
Where do eval cases come from?
The best ones come from production failures: every incident, bad outcome signal, or clustered failure becomes a case in the regression set. That is the loop part — a static set written at launch decays, while a set fed by live traffic tracks what your system actually faces.
Do I need special infrastructure to start?
No. Record your decisions (an SDK wrapper), write one rubric for your most embarrassing failure mode, and run a judge over the recorded events before each ship. The cadence matters more than the tooling — start with one gate this week.