Agent evaluation playbooks

Evaluation Dataset Design

An agent evaluation dataset should represent real task variation, meaningful failure modes, and held-out cases rather than only easy examples.

Evaluation Rubric Design

A useful rubric separates task-specific success dimensions and anchors them with examples that evaluators can apply consistently.

Evaluation Drift

A changing evaluation score can reflect data, evaluator, model, or task drift, so comparisons need matched versions and visible change context.

Tool Selection Evaluation

Tool-use evaluation checks selection, arguments, preconditions, ordering, and resulting state rather than rewarding only the final response.

Grounding Evaluation

Grounding evaluation checks whether material claims are supported by supplied context and whether missing evidence is handled honestly.

Handoff Assessment

Handoff evaluation checks whether an agent transfers a task with enough context, clear ownership, and an observable next action when it cannot continue safely.

Trajectory Evaluation

Trajectory evaluation examines the ordered states, tool calls, observations, and decisions that lead to an agent result, including recovery after a failed action.

Judge Calibration

Judge calibration aligns human or model evaluators on claim boundaries, scoring anchors, ambiguous cases, and reasons for a judgment before comparing agent results.

Evaluator Agreement

Evaluator agreement analysis separates genuine agent differences from inconsistent scoring by comparing labels, rationales, dimensions, and disagreement patterns.

Benchmark Contamination

Benchmark contamination checks whether evaluation items, prompts, references, or test artifacts leaked into agent inputs or training context and could inflate results.

Evaluation Reproducibility

Reproducible agent evaluation records preserve inputs, versions, execution conditions, scoring steps, and outputs so another reviewer can repeat and interpret a result.

Failure Taxonomy

A failure taxonomy gives agent evaluation consistent names for wrong outcomes, invalid paths, missing evidence, environment faults, and evaluator ambiguity without collapsing distinct causes.

Abstention Evaluation

Abstention evaluation checks whether an agent recognizes insufficient evidence, declines unsupported claims, asks for needed context, and still answers when the request is safely answerable.

Instruction Following Evaluation

Instruction-following evaluation checks whether an agent satisfies explicit constraints, priorities, formats, exclusions, and conditional requests while still completing the underlying task.

Retrieval Coverage Evaluation

Retrieval-coverage evaluation checks whether the context supplied to an agent contains the evidence needed for the task, across relevant, missing, distracting, and redundant sources.

Multi-Agent Coordination Evaluation

Multi-agent coordination evaluation checks role boundaries, message quality, shared state, dependencies, and recovery when several agents contribute to one task.

Cost Quality Frontier

Cost-quality evaluation compares agent quality with latency, token, tool, and review cost so a cheaper or faster path is not mistaken for an equivalent result.

Adversarial Evaluation Slice

Adversarial evaluation slices probe known weaknesses with misleading, conflicting, boundary, or pressure cases while preserving a clear task contract and safe interpretation.

Regression Gate Design

Regression-gate evaluation defines repeatable thresholds, slices, baselines, and review rules that determine when an agent change needs investigation before release.

Human Review Sampling

Human-review sampling selects representative and risk-focused agent outputs for people to inspect, balancing coverage, uncertainty, disagreement, and review cost.