Decide where the work belongs
Rank tool failure consequence
Describe the result of wrong tool, argument, order, or precondition handling. Give weight to state-changing calls, lost context, and failures that a final answer cannot expose. Keep harmless inefficiency separate from incorrect state.
Compare path coverage
Review single calls, dependent chains, parallel work, alternatives, unavailable tools, and recovery. Identify which cases distinguish similar tool schemas. Prefer a smaller set of diagnostic paths over many repeated happy-path calls.
Choose a score queue
Select cases, expected trace fields, owner, and review date. Defer lower-risk variations with a trigger. If the evaluator cannot observe state or arguments, prioritize instrumentation or evidence capture before scoring more traces.
What to carry forward
The priority output is a tool-use queue with consequence, confusion risk, path complexity, trace need, and owner. Start with cases where a wrong path could survive final-answer review. Keep low-risk efficiency cases separate from correctness failures before committing scarce evaluation capacity.
Technical background: Agent evaluation research on arXiv.
Keep the decision with the work.
Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.
Create an account See the product workflow