Aglet

Triage an Agent Evaluation Failure Taxonomy

A single “failed” label hides whether the agent, tool, task, evaluator, or environment caused the problem. Triage the taxonomy before collecting more examples. Define mutually useful categories, allowed multiple labels, and the evidence required for each label in the taxonomy.

Establish what is happening

  1. Define failure units

    State whether a label applies to task, step, trajectory, output, tool call, or evaluation record. Identify outcome, process, evidence, and environment dimensions. Keep symptom labels explicitly separate from suspected causes and ownership.

  2. Map representative cases

    Include wrong answer, invalid state, missed instruction, tool error, timeout, missing trace, ambiguous rubric, and valid alternate path. Record observed facts and hypotheses separately. Allow multiple dimensions only when their boundaries remain clear.

  3. Bound label use

    Specify how labels drive triage, prioritization, investigation, verification, or learning. Define unknown and not-applicable treatment. Do not force a causal category when the trace cannot establish one from observed evidence.

What to carry forward

The triage output is a failure taxonomy with units, dimensions, examples, evidence rules, and unknown handling. Stop when reviewers can label a case without guessing its cause. Keep suspected causes provisional until the path is investigated. Record the evidence owner and next check before routing cases.

Technical background: Agent evaluation research on arXiv.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow