Build a useful investigation brief
Trace case provenance
For each slice, record source, construction method, task context, redaction, label owner, and version. Identify duplicated prompts, near-duplicates, and examples that depend on hidden information. Preserve the reason each case belongs in the benchmark.
Compare task behavior
Contrast common and boundary cases with available real or production-like traces. Check input length, ambiguity, tool availability, state transitions, and outcome definitions. Identify where the benchmark allows a shortcut that a real task would not.
Test label and split quality
Review disagreements, invalid cases, contamination paths, and whether the holdout can reveal the intended failure. Re-run a small sample through the label process. State which dataset change would most improve the evidence.
What to carry forward
The investigation brief should link case provenance, task contrasts, labels, split policy, and uncovered failures. End with a dataset revision or named evidence gap. Do not claim production validity when the benchmark excludes important contexts or permits unrealistic shortcuts.
Technical background: Agent evaluation research on arXiv.
Keep the decision with the work.
Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.
Create an account See the product workflow