Aglet

Verify Agent Failure Labels Are Consistent

Verification tests whether taxonomy labels lead to reliable action. Review clear, borderline, and multi-cause examples with independent labels. A passing taxonomy makes important distinctions without pretending a thin trace proves the cause in the next evaluation and routing decision. Add evidence for each boundary.

Check whether the outcome improved

  1. Set label checks

    Require unit, observed evidence, outcome, process, tool, environment, evaluator, and unknown rules. Define neighboring categories, multiple-label treatment, and when suspected cause must remain provisional until evidence clearly distinguishes the label.

  2. Review varied cases

    Apply labels to clear, borderline, multi-cause, missing-trace, tool-fault, and valid-alternate-path examples. Compare labels and rationales. Inspect whether a shared category hides different ownership or next steps for independent taxonomy reviewers.

  3. Approve taxonomy scope

    State which task types, traces, evaluators, and decisions the labels support. Record examples, owner, version, and retest trigger. Mark classifications inconclusive when a material unit or evidence source is absent for that scope.

What to carry forward

The verification record should show label unit, evidence, examples, independent reasoning, unknown handling, and scope. Approve only the classification uses tested. Reopen it when task types, trace fields, rubric, or category boundaries change. Record a follow-up when ownership, causality, or evidence status remains disputed after independent review of the sample.

Technical background: Agent evaluation research on arXiv.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow