Aglet

Investigate a Product Experiment Result

A movement in an experiment metric can come from the treatment, selection, timing, instrumentation, or unrelated events. Investigation follows the result back to eligibility and exposure, then compares affected and unaffected paths. Keep the causal claim narrower than the metric whenever the design cannot support more.

Build a useful investigation brief

  1. Audit who was exposed

    Reconstruct eligibility, assignment, treatment receipt, comparison access, and exclusions for a sample of records. Look for users who crossed conditions or never saw the intended experience. Record implementation changes that occurred during the observation window.

  2. Compare behavior and guardrails

    Review the primary outcome beside task completion, support contacts, error paths, and downstream effects. Break the result down by relevant context without fishing for a favorable segment. Inspect individual cases that the aggregate conceals.

  3. Challenge causality

    Test timing, novelty, seasonality, concurrent releases, and measurement changes as alternatives. If the comparison is contaminated or the window is too short, state that limitation and identify the next design that could separate explanations.

What to carry forward

The investigation brief should link exposure records, metric definitions, guardrails, segment comparisons, and alternative explanations. End with a bounded causal interpretation or a request for a cleaner comparison. A positive movement alone does not prove durable user value or safe scale.

Technical background: IDEO design thinking guidance.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow