Aglet

How to verify job retry safety and final outcomes in context

Verification must prove that a retry either completes the intended logical job once or leaves an explicit recoverable failure. Exercise failures before and after side effects, inspect final records and attempt history, and confirm exhausted work is visible without unsafe automatic repetition.

Check whether the outcome improved

  1. Replay failures before and after effects

    Run the representative job with a controlled failure before its durable effect and another after the effect boundary. Allow the configured retry path, then compare attempt identifiers, record counts, values, timestamps, and final status. Keep external effects observable through safe test doubles or fixtures.

  2. Check retry classification and limits

    Exercise a recoverable error, a permanent validation error, and repeated failure through exhaustion. Confirm only intended classes retry, delays are recorded, and the final state explains what requires review. Verify a manual replay or operator action has clear ownership and remains idempotent.

  3. Inspect neighboring jobs and backlog

    Run an unrelated job while the representative job retries and confirm one poison pattern does not hide or duplicate other work. Check queue age, completion ordering where required, audit evidence, and customer-visible state. Record any capacity or isolation regression introduced by the fix.

What to carry forward

Classify retry behavior as safe, unsafe, unchanged, or inconclusive with attempt, state, and exhaustion evidence attached. It passes only when logical effects occur once, intended failures retry, permanent failures stop clearly, and recovery remains reviewable without data ambiguity.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow