Aglet

Triage shutdown interruptions by checkpoint and recovery owner

Stopping a worker can be safe at one phase and ambiguous at another. Triage one interrupted job by recording the shutdown signal, drain decision, active step, last durable checkpoint, process exit, and recovery owner. Compare a safe-point stop with a mid-step stop before deciding whether work was lost or repeated.

Establish what is happening

  1. Capture the shutdown sequence

    Record worker ID, signal time, drain or stop mode, job claim, phase, checkpoint, side effects, process exit, requeue, restart claim, and final state. Keep the worker’s last durable record separate from its last log line.

  2. Compare safe and mid-step stops

    Inspect a job stopped before its next effect and one stopped during a known effect or wait. Note whether the job is requeued, marked interrupted, or left claimed. Preserve the phase and recovery owner so shutdown timing is not treated as one generic failure.

  3. Check for lost or repeated work

    Reconcile item outcomes, output identities, claims, and side effects before and after restart. Identify any item with no owner or more than one effect. If a worker exited after a safe commit, distinguish legitimate replay from duplicate processing.

What to carry forward

Return shutdown signal and exit timeline, active phase, checkpoint, claim owner, recovery path, and effect reconciliation. Classify the scope as drain, checkpoint, requeue, ownership, or terminal reporting. Keep loss or duplication unresolved when the last process record cannot be correlated.

Technical background: Celery documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow