Aglet

Turn Retrieval Gaps Into Better Agent Evaluation

Coverage cases teach whether evaluation sees the evidence path behind an answer. Preserve required claims, retrieved sources, omission, generation use, and decision impact. Turn the case into a reusable slice that keeps retrieval quality distinct from fluent response quality for evidence reviewers.

Keep the lesson for the next incident

  1. Save the coverage case

    Store task, evidence map, query, corpus, filters, ranked sources, context, answer, support labels, omission, and final interpretation. Include a case where retrieval had evidence but the agent ignored it, and one where evidence was absent.

  2. Improve coverage practice

    Add multi-hop, stale, conflicting, distractor, empty, provenance, and context-limit checks to evaluation plans. Require a retrieval-versus-generation label. Explain how the change addresses the exact omission or ignored passage in review.

  3. Watch for hidden gaps

    Set a signal such as fluent answers with absent required claims, scores rising after removing hard cases, missing source versions, context truncation, or retrieval and generation failures sharing one label. Assign an owner to sample evidence maps and revise checks when it appears.

What to carry forward

The learning record should connect the evidence gap to the revised retrieval slice or provenance practice and recurrence signal. State which required claim or source check became visible. Keep the lesson tied to the tested corpus and task evidence rather than treating retrieval score as answer quality.

Technical background: Ragas evaluation documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow