Aglet

Triage an Agent Trajectory Evaluation

A final answer hides the route an agent took to reach it. Triage the trajectory contract before judging quality: define which transitions matter, which actions are acceptable, and where missing trace evidence prevents a fair decision. Keep outcome and path signals distinct.

Establish what is happening

  1. Define the trace contract

    List required state fields, observations, tool calls, arguments, returns, decision points, retries, and completion conditions. Identify state changes that need snapshots. Separate a valid alternate path from an incomplete trace.

  2. Map trajectory variants

    Include direct success, dependent calls, failed tools, recovery, loops, partial completion, and an answer that is correct by chance. Record task fixture, available tools, and expected state after important transitions.

  3. Bound the score

    State whether the review measures process validity, efficiency, recovery, final outcome, or a combination. Mark hidden state, absent observations, and unrecorded side effects as evidence gaps. Do not infer planning quality from trace length without state evidence.

What to carry forward

The triage output is a trajectory contract with trace fields, transition rules, acceptable paths, and score limits. Stop when reviewers can reconstruct a case without hidden context. Keep results provisional when state changes or observations are missing from the recorded trajectory and state log.

Technical background: Agent evaluation research on arXiv.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow