Your test set is rotting: continuous evals vs static benchmarks
July 12, 2026 · the looptail team
The eval set you wrote at launch had one flaw baked in: it measures the system you launched. The one you run today — three model swaps, forty prompt edits, and two new tool integrations later — is a different system facing different traffic. Your greens are telling you about the past.
Silent decay is the dangerous kind
A rotting test set doesn't fail loudly; it passes loudly. Scores stay green because the questions froze while the system and its traffic moved. The new failure modes — the ones born from March's model swap and April's prompt hotfixes — simply aren't in the set, so nothing measures them. Teams discover the gap the expensive way: an incident that no eval predicted, followed by the realization that the eval couldn't have predicted it.
What "continuous" concretely means
- Score live traffic, not just fixtures — sampled if volume demands (the cost math is friendlier than you think).
- Rubrics over metrics: versioned criteria encoding your policies, applied by a judge model you choose — so "fluent but violates the refund policy" fails.
- Failures feed the set. Every clustered production failure becomes a regression case. The set grows where reality pushes.
- Verdicts join the record. Each evaluation lands next to the decision it judged, in an append-only signed trail — so "we monitor our system" is provable, not asserted.
The migration path is shorter than it looks
You don't rewrite your eval stack; you re-point it. Keep your benchmark suite for model selection — tools like LangSmith serve that development loop well. Then put recording in the production path, write one rubric for your scariest policy, and run it on this week's real traffic. The first time a green benchmark and a failing rubric disagree, you'll know which one was measuring your system.
FAQ
Why do static test sets decay?
Three clocks run against them: your system changes (models swapped, prompts hotfixed), your traffic changes (new user segments, new phrasings, adversarial patterns), and your standards change (policies updated, new failure modes discovered). A fixed set measures none of that movement — it keeps scoring the world as it was on the day it was written.
How fast does the rot set in?
As fast as your rate of change. A team swapping models quarterly and hotfixing prompts weekly can find within a quarter that its eval set no longer covers its top three production failure modes — they emerged after the set was frozen. The decay is silent: scores stay green because the set stopped asking hard questions.
What is continuous evaluation?
Scoring live production traffic — sampled or complete — against versioned rubrics that encode your current standards, with failures clustered and fed back into the regression set. The eval set becomes a living artifact maintained by production itself rather than a snapshot.
Do benchmarks have any role left?
Yes: model selection. Public benchmarks are useful for comparing candidate models before you adopt one. What they cannot do is tell you whether your system, on your traffic, meets your standards this week — that requires evals derived from your own production.