Aglet

Prioritize Agent Instruction Following Cases

Instruction tests should start where a polished answer can conceal a decisive miss. Prioritize state-changing requirements, exclusions, conditional branches, and output formats used downstream. Keep harmless optional-detail differences out of the first queue before assigning a release quality score. to any model.

Decide where the work belongs

  1. Rank constraint consequence

    Describe what a missed action, order, exclusion, format, or condition would change. Give weight to irreversible state, downstream parsing, user trust, and requirements whose violation is difficult to notice after delivery.

  2. Compare instruction difficulty

    Review single, chained, conditional, conflicting, format-bound, and incomplete requests. Identify which cases distinguish literal compliance from useful interpretation. Prefer a small set of diagnostic constraints with visible pass criteria for independent reviewers.

  3. Choose a compliance queue

    Select case, obligation, severity, owner, and review date. Defer low-impact wording differences with a trigger. If priority between instructions is unclear, prioritize clarifying the contract before scoring outputs for review.

What to carry forward

The priority output is an instruction queue tied to consequence, silent-failure risk, dependency, and visibility. Start where one missed constraint could invalidate the result. Keep optional style preferences separate from required task behavior. Record a follow-up for unresolved priority, format, or branch rules before publishing the next comparison.

Technical background: OpenAI evaluation guidance.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow