Aglet

Investigate a flaky test with reproducible runs

Investigation should turn intermittent observations into a set of testable explanations. Reproduce under controlled differences, preserve seeds and fixtures, and distinguish a test harness issue from nondeterministic product behavior.

Build a useful investigation brief

  1. Control one variable

    Repeat the test with fixed revision, fixture, seed, ordering, clock, browser or runtime, and parallelism. Change one condition at a time and record whether the failure signature moves or disappears.

  2. Inspect asynchronous boundaries

    Review waits, jobs, database visibility, cache state, network responses, shared resources, and cleanup around the failing assertion. Treat timing as a hypothesis until a controlled comparison supports it.

  3. Compare product evidence

    Use logs, state transitions, screenshots, or request records from passing and failing runs to ask whether the product behaved differently. A stable product with unstable assertion points to a different repair path.

What to carry forward

The brief should identify the strongest reproducible condition, failure signature, and alternative explanations, with run artifacts and confidence. Conclude harness-related, product-related, environment-related, or unresolved, and state the smallest next experiment. Compare passing and failing runs for order, seed, timing, shared state, network boundaries, data setup, and cleanup; keep each cause provisional until a discriminating observation exists.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow