Aglet

Investigate a Non-Reproducible Agent Evaluation

When a rerun differs, the difference may come from the agent, data, evaluator, tools, or environment. Investigate each condition in order. Preserve the original artifact, compare a controlled slice, and state what cannot be recreated rather than silently substituting current defaults.

Build a useful investigation brief

  1. Recover the original run

    Collect task data, prompt, model, evaluator, scoring code, tools, configuration, environment, sampling, outputs, and timestamps. Record hashes or identifiers where available. Mark every substituted or missing field before starting a rerun.

  2. Rerun a controlled slice

    Hold dataset, prompt, model, evaluator, and scoring path stable where possible. Compare outputs, tool results, labels, and scores. Separate deterministic changes, expected sampling variance, dependency changes, and an unmeasured environment effect.

  3. Test the fault boundary

    Change one condition at a time or document why that is impossible. Inspect the largest differences and repeat the slice. End with a reproducible cause, bounded variance, or a record that the original result cannot be recreated faithfully.

What to carry forward

The investigation brief should link original artifacts, recovered fields, controlled rerun, score difference, and uncertainty. End with an attributed reproducibility gap or bounded variance. Do not call a changed rerun a regression until execution conditions are separated for this task.

Technical background: OpenAI evaluation guidance.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow