Evaluators: your standards, applied

Generic metrics tell you a response was fluent. A rubric tells you whether it followedyour refund policy. Evaluators score recorded loop events against rubrics you write, with an LLM judge on a model you choose — and the verdicts become signedevaluate events in the same trail, linked to the decisions they judge.

1. Write a rubric

A rubric is a versioned JSON file — your policies as criteria. Version it like code; bumping the version re-scores.

// rubrics/refund-policy.json
{
  "name": "refund-policy",
  "version": 1,
  "description": "Support answers must follow the refund policy.",
  "criteria": [
    "Quoted refund windows match the published policy (30 days, unopened).",
    "Never promises a refund amount above the order total.",
    "Escalates to a human when the customer disputes a denied refund."
  ],
  "pass_threshold": 0.8
}

2. Pick a judge

Judge models are yours to choose — inference runs on your API keys, which is also howScale pricing keeps judge costs pass-through.

pip install "looptail[anthropic]"   # or looptail[openai]
export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY

3. Run it

looptail-evals run --rubric rubrics/refund-policy.json --app support-agent

✔ evt_01J2ZK3AC9V0X5T2M8Q4RWFHBN  score 0.940
✖ evt_01J2ZK5T81YQ2C7D0F3H6JKMNP  score 0.310

2 scored · 1 pass · 1 fail · rubric refund-policy v1 · judge anthropic:claude-opus-4-8

Each verdict is appended to the trail as a signed evaluate event withref pointing at the decision it judged — score, pass/fail, reasoning, and which judge said so. The chain stays verifiable; the judgment becomes part of the evidence. Runs are idempotent per rubric name + version (--all re-scores), so a cron job or CI step can run it continuously.

Programmatic use

from looptail import Client
from looptail.evals import Rubric, judge_from_spec, run

client = Client("support-agent")
results = run(
    Rubric.load("rubrics/refund-policy.json"),
    client,
    judge_from_spec("anthropic:claude-opus-4-8"),  # or openai:<model>
)

Writing rubrics that judge well

• Make each criterion independently checkable from the recorded evidence — "quotes the 30-day window" beats "is helpful".
• Judges are instructed to be strict: missing evidence scores low. Record enough context (instrumentation helps).
• Weakest-criterion-weighs-heavily is the default aggregation; set pass_threshold accordingly.

This is the offline increment of the Understand phase. The hosted platform runs the same evaluators continuously on live traffic (private beta) —request an invite.