Aglet

Document worker shutdown, drain, and restart ownership

The lasting lesson is a shutdown contract that names what the worker drains, what it checkpoints, and what a replacement worker may claim. Record the recovery boundary for each side-effect phase and how uncertain work is held for reconciliation. A process exit alone cannot define job outcome.

Keep the lesson for the next incident

  1. Write the shutdown contract

    Define idle drain, safe-point stop, mid-step interruption, checkpoint durability, claim release, requeue, and terminal state. State which effects may finish and how a replacement proves ownership before resuming a job.

  2. Retain restart fixtures

    Keep idle drain, pre-effect, post-commit, mid-step, and worker-replacement cases with expected claims, checkpoints, and effects. Re-run them after changing signal handling, worker lifecycle, job phases, transaction scope, or requeue logic.

  3. Watch recurrence signals

    Review exited workers with active claims, restarted jobs whose checkpoint lags their effects, and duplicate output identities after shutdown. Reconcile before replaying. Assign review when a new phase or worker mode lacks a shutdown boundary.

What to carry forward

Preserve shutdown and restart ownership rules with phase checkpoints, requeue semantics, and effect reconciliation. Recurrence checks should connect process exit to durable claims and outcomes. Reopen review when worker lifecycle, job phase, transaction scope, signal mode, or recovery changes what interruption means.

Technical background: Celery documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow