Aglet

Triage a webhook retry storm

A retry wave may begin with one failing dependency and grow through several retry owners. Freeze a short window and record event IDs, response classes, attempt times, endpoint latency, and queue state. Keep sender retries separate from local worker retries and new events.

Establish what is happening

  1. Capture the first wave

    Record endpoint, event and delivery IDs, response status, attempt number, arrival time, response time, and local queue state. Redact payload and secret. Note whether the receiver had durably recorded the event before the next attempt.

  2. Trace retry ownership

    Follow a failed response through sender schedule, endpoint handler, local worker, and downstream retry. Compare timing and identity at each hop. A second attempt may be sender-driven, locally scheduled, or a fresh event with a similar payload.

  3. Bound the pressure

    Group attempts by endpoint, event type, client or handler revision, response class, age, and effect state. Separate outage, status mapping, capacity, and duplicate claims. Preserve essential work and unknown effects for review.

What to carry forward

Triage ends with a bounded wave, retry owners, event identity, receipt state, affected work, and evidence gap. Stop synchronized local retries while the path is uncertain. Route sender policy, receiver capacity, and handler questions separately.

Technical background: Twilio documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow