Aglet

Investigate an Agent Abstention Error

Abstention errors often sit between answer quality and evidence quality. Recreate what the agent could see, split the request into answerable parts, and compare the chosen response with safe alternatives. Preserve the user next action as part of the outcome.

Build a useful investigation brief

  1. Reconstruct the boundary

    Record request, intent, context, retrieval results, tool state, prompt, response, confidence language, and expected answerability. Mark missing or conflicting evidence. Preserve versions and the exact response shown to the user.

  2. Compare response choices

    Test answer, qualified answer, clarifying question, useful partial response, and decline against the evidence. Identify unsupported claims, unnecessary refusal, hidden ambiguity, or a missing next step. Keep response length separate from evidence handling.

  3. Test competing causes

    Replay with complete, missing, conflicting, and clarified context. Check whether the response changes at the intended boundary. State whether the fault is evidence detection, intent interpretation, response policy, or evaluator ambiguity.

What to carry forward

The investigation brief should link request intent, evidence boundary, response choice, alternatives, replay, and user next action. End with a localized abstention error or evidence gap. Do not call a cautious answer wrong merely because it is shorter for the stated request and context under review.

Technical background: Google DeepMind evaluation research.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow