Judge models: what continuous evaluation actually costs
July 12, 2026 · the looptail team
"We can't afford to evaluate everything" is the most common reason continuous evaluation dies in planning. Let's actually run the numbers — they're better than you've been told, and the levers that control them belong in your hands.
The unit economics
One rubric evaluation is one judge call: the recorded decision (typically 500–2,000 tokens), your criteria (a few hundred), and a structured verdict out (a few hundred). Call it ~2–3k tokens round trip. Even on a frontier judge that's cents; on a mid-tier judge it's fractions of a cent. Multiply by your monthly decision volume and the naive number can still look scary — which is where the levers come in.
Three levers, in order of power
- Sampling strategy. Judge 100% of high-stakes kinds; always judge the failure-adjacent slice (escalations, retries, negative outcome signals — it's where the information lives); sample the routine bulk at 1–10%. Most teams keep >80% of the signal for <20% of the naive cost.
- Judge-model choice. Judging with explicit criteria is easier than open-ended generation; mid-tier models handle most rubrics. Validate against a human-labeled sample of ~100 decisions, then spend frontier tokens only where nuance demands it.
- Rubric design. Tight, evidence-checkable criteria produce short, cheap verdicts. Vague criteria produce long reasoning and noisy verdicts — expensive twice.
Compare it to the alternative
The honest baseline isn't zero — it's the cost of not knowing: one incident that evaluation would have flagged, one enterprise deal stalled on "how do you monitor your AI?", one postmortem that takes two weeks because nothing was scored at the time. Continuous evaluation at a few hundred dollars a month is the cheap side of that ledger. And the verdicts compound: each one lands in thesigned trail next to the decision it judged, turning "we monitor our system" into something you can prove.
One structural note: whoever sells you evaluation has opinions about your judge spend. We've priced ourselves out of that conflict — judge inference onScale is pass-through on models you pick. Your evaluation strategy should be an engineering decision. (It's also whylooptail-evals runs on your own API keys, starting today, free.)
FAQ
How much does LLM-as-judge evaluation cost per decision?
A rubric evaluation is roughly 1–3k input tokens (the recorded decision plus your criteria) and a few hundred output tokens. On current frontier-model pricing that lands in the fractions of a cent to a few cents per decision, and an order of magnitude lower on smaller judge models. The real number depends on the model you pick — which is why the model should be your choice, not the vendor’s.
Do I need to judge every decision?
Rarely. High-stakes kinds (refund approvals, medical or credit decisions) may justify 100% coverage; for the rest, stratified sampling — always judge failures-adjacent traffic like escalations and retries, sample the routine bulk at 1–10% — preserves most of the signal at a fraction of the cost.
Should the judge be a bigger or smaller model than the system?
Judging is usually easier than generating: the judge gets explicit criteria and the completed record. Many rubrics work well on mid-tier models. Reserve frontier judges for nuanced policy calls, and validate any judge against a small human-labeled sample before trusting it at scale.
Why does Looptail pass judge inference through at cost?
Because marking up judge tokens creates an incentive to push you toward expensive judges and over-evaluation. On the Scale plan, judge inference runs on models you choose and bills at cost-plus a small handling margin — evaluation strategy stays an engineering decision, not a revenue lever.