Aglet

Verify an Agent Evaluation Dataset Before Scoring

Verification protects an agent score from being a property of the dataset alone. Review task inputs, allowed actions, labels, splits, and edge cases before comparing systems. A verified set can still be narrow, but its scope and blind spots must be explicit.

Check whether the outcome improved

  1. Set dataset checks

    Require task goal, context, allowed tools, expected answer or state, stopping condition, provenance, label rule, version, and split assignment. Add cases for ambiguity, missing evidence, recovery, and valid alternative paths when they matter.

  2. Review cases and labels

    Sample each slice and inspect duplicates, invalid inputs, label rationale, and context completeness. Compare difficult and ordinary cases. Record disagreement and a known failure the set would not detect during scoring.

  3. Approve the scoring boundary

    State which comparisons the dataset supports, how holdout access is controlled, and what requires another slice. Record the owner, version, and review date. Keep a result inconclusive when task or label quality blocks interpretation for this version.

What to carry forward

The verification record should show task checks, case coverage, label review, split protection, and scoring limits. Use results only for the dataset version and behavior tested. Add a follow-up when a blind spot could change the system ranking.

Technical background: Agent evaluation research on arXiv.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow