Keep the lesson for the next incident
Save the dataset case
Store the task contract, source or construction rationale, slice, label decision, counterexample, and evaluation impact together. Include a case that looked representative until a missing context changed its meaning when the slice was reviewed.
Improve benchmark maintenance
Add provenance, variation, duplicate, label-rationale, and holdout checks to the dataset release process. Require a review of critical slices before publishing a new version. Explain how the change addresses the observed confidence gap.
Watch for easy-set drift
Set a signal such as rising scores with unchanged failure reports, repeated prompt patterns, or new cases from only one source. Assign an owner to sample the benchmark and revisit its design when the signal appears.
What to carry forward
The learning record should connect the dataset blind spot to the revised maintenance practice and recurrence signal. State which slice or label rule changed. Keep the lesson tied to the task contract instead of treating one benchmark as a universal measure.
Technical background: Agent evaluation research on arXiv.
Keep the decision with the work.
Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.
Create an account See the product workflow