Establish what is happening
Define the tool contract
List task goal, available tools, schemas, preconditions, required arguments, permission boundary, expected state, and completion rule. Identify actions that are reversible, ambiguous, or invalid. Separate tool correctness from final wording.
Map path variation
Include single-tool, chained, parallel, ambiguous, unavailable, and recovery cases. Record whether several tool paths are valid. Add expected dependency and argument checks so a short path is not automatically considered better.
Bound the score
State what a selection score measures and what it cannot infer about planning, user value, or hidden state. Identify trace fields needed for review. Keep cases with missing tool outcomes separate from true selection failures.
What to carry forward
The triage output is a tool contract with path types, argument rules, state expectations, trace fields, and score boundary. Stop when reviewers can judge a call without guessing. Keep the result provisional when tool outcomes or preconditions are not recorded during release review.
Technical background: Agent evaluation research on arXiv.
Keep the decision with the work.
Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.
Create an account See the product workflow