Aglet

Verify a flaky test is reliable in regression runs

Verification should show that the evidence source is trustworthy under the condition that exposed flakiness. Repeat enough comparable executions to test the explanation, while avoiding an unsupported claim that an intermittent problem can never return.

Check whether the outcome improved

  1. Replay the trigger

    Use the same seed, fixture, ordering, concurrency, clock, browser or runtime, and dependency conditions from the failing runs. Record any unavailable condition and how it limits the comparison.

  2. Inspect the assertion path

    Confirm the test waits for the intended state, cleans up shared resources, and still exercises the customer behavior. A passing test that no longer reaches the original assertion is not recovery.

  3. Review repeated outcomes

    Compare the corrected runs with prior failures and any controlled baseline. Record passes, failures, timing, and product evidence, then keep the result qualified if recurrence coverage is incomplete.

What to carry forward

Verification should say reliable under the reviewed conditions, still flaky, product failure exposed, or inconclusive. Close only when the original behavior remains covered and the correction addresses the observed instability. Repeat the smallest reproduction across fresh and repeated runs, then confirm the test detects a real regression and remains useful when the suspected timing or ordering changes.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow