Aglet

Turn Human Review Sampling Into Better Agent Evaluation

Human review cases teach whether people are inspecting the right outputs. Preserve the population frame, selection reason, reviewed labels, and decision impact. Turn the lesson into a maintained sampling plan that balances representative coverage with purposeful risk discovery for decisions.

Keep the lesson for the next incident

  1. Save the sampling case

    Store population frame, window, task mix, selection method, strata, exclusions, reviewer context, labels, disagreement, nonresponse, and conclusion. Include a targeted sample that found a failure and a representative sample used for prevalence.

  2. Improve review practice

    Add frame ownership, selection metadata, representative and targeted strata, reviewer context, label calibration, and nonresponse tracking to evaluation plans. Explain how the change addresses the missing group or overgeneralized result.

  3. Watch for sampling drift

    Set a signal such as repeated convenience examples, recent outputs dominating review, protected slices missing, disagreement unlogged, or targeted findings reported as prevalence. Assign an owner to inspect samples and revise the plan when it appears.

What to carry forward

The learning record should connect the sampling gap to revised frame, selection, or reviewer practice and recurrence signal. State which population or label became visible. Keep the lesson tied to the tested review question rather than treating human sampling as automatically representative.

Technical background: Google DeepMind evaluation research.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow