The Improve loop

Observability tells you the agent failed. The Improve loop turns those failures into a tested, signed-off change — and records every step, so the trail answers not justwhat did the agent do but why did we change it, what did we test, and who approved it.

1. Cluster failures into issues

Failing verdicts from your evaluators group by rubric and weakest criterion into tracked issue events. Clustering is deterministic on purpose — an issue reads as "refund-policy v2, criterion 1 keeps failing", with no embeddings or opaque similarity to argue with in a postmortem.

looptail-issues cluster --app support-agent

● refund-policy/v2/criterion-1  ×14  refund-policy v2: criterion 1 failing

1 issue(s) updated · run looptail-improve propose --issue <key>

Re-running after new failures appends an updated issue event with the same key and a cumulative count. Nothing is ever re-counted or edited in place.

2. Propose a patch

An LLM proposer reads the current prompt, the failing criterion, and the recorded failure examples, then drafts the smallest change that plausibly fixes them. The draft lands under .looptail/proposals/ — your prompt file is untouched.

looptail-improve propose --app support-agent \
  --issue refund-policy/v2/criterion-1 \
  --prompt-file prompts/support.txt

proposal prop_01J3… → .looptail/proposals/prop_01J3….txt
rationale: failures show answers promising refunds outside the 30-day window…

3. Replay a regression set

Before anything ships, the proposal runs against a regression set withyour runner — the same code path your app uses — and every output is judged against the rubric. Results are recorded, pass or fail.

looptail-improve replay --app support-agent --proposal prop_01J3… \
  --cases regression.jsonl \
  --runner myapp.replay:run \
  --rubric rubrics/refund-policy.json

✔ 24/24 passed

The runner is any Python callable (prompt_text, case_input) → output, imported by path. Cases are JSONL, one {"input": …} per line.

4. Canary and approve — gated

canary records a partial rollout; approve --apply writes the proposal over the prompt file. Both refuse without a passing replay on record. --force overrides the gate, and the override itself is recorded in the event body — permanently.

looptail-improve canary  --app support-agent --proposal prop_01J3… --percent 10
looptail-improve approve --app support-agent --proposal prop_01J3… --apply

applied: prop_01J3… by max

The approve event names who decided, what replay evidence backed the decision, and which file changed. Six months later, that is the difference between "we think we tested it" and evidence.

What lands in the trail

Step	Event	Body highlights
cluster	`issue`	stable key, cumulative count, recent example refs
propose	`improve`	proposal id, rationale, proposer model, files
replay	`improve`	rubric + version, judge, passed/failed, per-case scores
canary	`improve`	percent, ref to the replay it rests on
approve	`approve`	who decided, replay evidence, applied or not, forced or not

Every event is hash-chained and Ed25519-signed like the rest of the trail —npx @looptail/cli verify covers your improvement history too.

Shown here with the Python CLI; the same loop ships in@looptail/sdk (propose/replay/canary/approveand npx @looptail/cli improve) at full parity. The hosted platform adds it on live traffic with a review UI (private beta) —request an invite.