Aglet

Investigate a Retrieval Coverage Failure

Retrieval investigation needs the task’s evidence map and the exact context the agent received. Compare required claims with retrieved passages, then inspect answer use. Preserve corpus, filters, ranking, and provenance so the failure is reproducible for a reviewer using the same corpus.

Build a useful investigation brief

  1. Map required evidence

    Record task, claims, entities, constraints, source types, query, corpus version, filters, ranked results, context window, and answer. Mark each requirement supported, partial, redundant, conflicting, or absent for the tested request.

  2. Trace the omission

    Identify the first missing or weakly ranked passage and check whether a relevant passage was retrieved but ignored. Inspect query wording, filters, chunking, ranking, limits, source freshness, and context truncation as competing causes.

  3. Test recovery paths

    Rerun with query, ranking, corpus, and context changes one at a time. Compare coverage and answer support. State whether the fault is source absence, retrieval selection, context delivery, generation use, or evaluator labeling.

What to carry forward

The investigation brief should link evidence map, retrieval context, ranked sources, omission, recovery, and answer use. End with a localized coverage failure or evidence gap. Do not call retrieval successful because a related passage appeared when a required claim remained unsupported in this run.

Technical background: Ragas evaluation documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow