Aglet

Turn Abstention Findings Into Better Agent Evaluation

Abstention cases reveal whether evaluation rewards certainty or useful honesty. Preserve the request, evidence boundary, response choice, and user effect. Turn the lesson into a reusable case that keeps unsupported confidence and unnecessary refusal visible in future evaluation runs later.

Keep the lesson for the next incident

  1. Save the abstention case

    Store request, intent, context, retrieval, response, claims, evidence labels, next action, score, and final interpretation. Include a safe partial answer and a refusal that lacked a needed clarification so the boundary is concrete.

  2. Improve boundary practice

    Add answerability slices, conflicting-context cases, clarification rules, useful partial-help checks, and unsupported-claim review to the evaluation plan. Explain how the change addresses the exact overconfidence or over-refusal observed in review.

  3. Watch for confidence drift

    Set a signal such as precise claims without support, refusals on complete evidence, clarifying questions that never identify missing information, or partial answers without a next action. Assign an owner to sample cases and revisit checks when it appears.

What to carry forward

The learning record should connect the response boundary to the revised case or evidence practice and recurrence signal. State which answer, qualify, clarify, or decline rule changed. Keep the lesson tied to the tested context and request types rather than prescribing one refusal rate.

Technical background: Google DeepMind evaluation research.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow