Aglet

Triage an Agent Evaluation Regression Gate

A gate can block harmless variance or miss a real regression if its baseline and slices are vague. Triage the gate contract before setting a threshold. Separate signal detection from root-cause investigation and preserve a path for inconclusive results that keeps the evidence boundary visible.

Establish what is happening

  1. Define the baseline

    Record model, prompt, dataset, evaluator, scoring code, task mix, and versioned reference. State quality dimensions, protected slices, tolerance, and what counts as a meaningful change for each recorded gate run.

  2. Map gate outcomes

    Include pass, fail, borderline, missing evidence, evaluator change, slice shift, and infrastructure failure cases. Define who reviews each outcome and whether the agent result or gate itself needs first investigation.

  3. Bound release use

    Specify which changes, task families, and decisions the gate supports. Define override evidence and follow-up without silently bypassing the threshold. Do not use one aggregate score when a protected slice can regress.

What to carry forward

The triage output is a regression-gate contract with baseline, slices, dimensions, threshold, variance, outcomes, and review path. Stop when a crossing has a defined next action. Keep the gate provisional when baseline or evaluator versions are unknown. Record who owns the next check before a release decision.

Technical background: OpenAI evaluation guidance.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow