Aglet

Learn from webhook retry storms

The durable lesson is a clear division between sender retries, receiver retries, and downstream work. Preserve the first failure, response class, receipt state, and retry wave that mattered. Document finite containment and effect identity without promising a fixed sender schedule.

Keep the lesson for the next incident

  1. Document retry ownership

    Record which layer retries, response classes, attempt limits, backoff evidence, receipt state, deduplication, and permanent disposition. Define the safe action for an exhausted event and an unknown write. Keep sender and receiver attempts separate.

  2. Keep storm fixtures

    Retain one failure, concurrent burst, recovery, duplicate-after-receipt, invalid event, local requeue, and restart cases. Store expected attempts, IDs, receipt, and effects. Include the original amplification shape for future response changes.

  3. Review pressure signals

    Watch attempt growth, synchronized retries, queue age, pre-receipt failures, duplicate effects, and status mapping by endpoint. Assign an owner and threshold. Close the follow-up only when shared handlers exercise finite retry and identity checks.

What to carry forward

Close learning with retry ownership, response rules, pressure fixtures, effect identity, and amplification signals. Keep sender limits visible. The useful outcome is a finite, reviewable recovery path when delivery pressure rises again. Revisit the retry contract when either the sender schedule or worker retry policy changes; overlapping retry layers can multiply attempts.

Technical background: Twilio documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow