Aglet

Verify Retrieval Coverage Before Judging Answers

Verification protects the boundary between retrieval and generation. Apply coverage checks to direct, multi-source, noisy, stale, conflicting, and no-result cases. A passing retrieval result makes required evidence available; it does not guarantee the final answer used it in practice under the tested corpus.

Check whether the outcome improved

  1. Set coverage checks

    Require evidence map, source provenance, corpus version, query, filters, ranking, context limit, required claims, omission rule, conflict rule, and downstream separation. Define sufficient coverage for single and multi-source tasks in the tested corpus.

  2. Review context slices

    Apply checks to complete, partial, redundant, noisy, conflicting, stale, and empty retrieval results. Inspect ranked sources and context delivered to the agent. Record a case where evidence was present but generation still failed.

  3. Approve retrieval scope

    State which task families, corpora, source versions, and evidence needs the result supports. Record owner, reviewer, date, and retest trigger. Mark coverage inconclusive when provenance or required claims cannot be checked.

What to carry forward

The verification record should show evidence map, sources, provenance, ranked context, omissions, conflicts, and separation from generation. Approve only the coverage behavior observed. Reopen it when corpus, query, ranking, context limits, or task evidence needs change before judging new outputs from that retrieval path after corpus review.

Technical background: Ragas evaluation documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow