Aglet

Turn Contamination Findings Into Better Benchmarks

Contamination reviews teach whether evaluation hygiene is part of measurement quality. Preserve the item history, controlled comparison, and qualification decision. Turn that case into a repeatable benchmark maintenance practice that keeps exposure visible without discarding useful evidence during future reviews.

Keep the lesson for the next incident

  1. Save the exposure case

    Store provenance, access history, model version, exposed and hidden scores, overlap evidence, controlled rerun, and final claim treatment. Include a clean item path and an uncertain path so future reviewers see the boundary.

  2. Improve benchmark practice

    Add ownership, versioned item records, hidden holdouts, fresh task refreshes, retrieval exclusions, and exposure review to the evaluation plan. Explain how the change addresses the exact leakage path or provenance gap found.

  3. Watch for score inflation

    Set a signal such as sudden gains on public items, near-verbatim answers, hidden-slice drops, missing item provenance, or score movement after retrieval exclusion. Assign an owner to inspect the signal and refresh checks when it appears.

What to carry forward

The learning record should connect the exposure path to the revised provenance or holdout practice and recurrence signal. State which claim was qualified or restored. Keep the lesson tied to benchmark lifecycle and observed controls rather than assuming every high score is contaminated.

Technical background: Google DeepMind evaluation research.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow