Aglet

Investigate an Agent Tool-Selection Failure

Tool failures need the whole trajectory, not a screenshot of the final reply. Reconstruct what the agent knew before each call, compare its choice with valid paths, and inspect state changes. Keep model uncertainty separate from tool schema or environment defects.

Build a useful investigation brief

  1. Reconstruct the decision point

    Record prompt, available tools, schemas, context, prior state, selected call, arguments, and expected preconditions. Capture what the tool returned and what the agent did next. Preserve the exact version and task fixture.

  2. Compare valid paths

    Identify alternative calls, required order, omitted arguments, and dependency violations. Check whether a different path would have achieved the outcome with less risk or cost. Record cases where multiple choices were valid and should not be marked failures.

  3. Test the fault boundary

    Replay the task with one schema, state, or context change at a time. Inspect whether the failure persists. Consider tool availability, ambiguous instructions, stale state, and evaluator blind spots before assigning the cause to the agent.

What to carry forward

The investigation brief should link the decision point, available tool contract, selected path, alternatives, state transition, and replay. End with a localized tool failure or evidence gap. Do not label a valid alternative as wrong merely because it differs from an expected path.

Technical background: Agent evaluation research on arXiv.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow