Aglet

Triage a flaky test without dismissing the failure

Flakiness is a reliability problem in the evidence itself. Start by preserving the failing assertion and the conditions around it, then compare passing and failing runs. Keep intermittent behavior separate from a one-time failure or a deterministic product defect.

Establish what is happening

  1. Name the failing check

    Record test file, example, assertion, suite, runtime, browser or device, seed, parallelism, timing, and relevant fixture or data state. Preserve the first failure output without copying secrets or unrelated logs.

  2. Compare adjacent runs

    Place passing and failing executions side by side and note code revision, dependency state, environment, ordering, clock, network, and resource conditions. Mark each difference as observed or merely suspected.

  3. Bound product exposure

    Decide whether the flaky check covers a customer path, release gate, or only internal tooling. A noisy test can still hide a real regression; keep the covered behavior and evidence quality visible.

What to carry forward

Triage should leave a reproducible test identity, failure signature, run context, and product boundary. Escalate when a customer path or release decision is affected; otherwise classify the evidence gap without dismissing the failing run. Preserve the failing test name, run identifier, failure output, environment, and whether reruns changed the result; a single green rerun does not explain the first failure.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow