Build a useful investigation brief
Prepare three phase fixtures
Create sanitized jobs that run through a short phase, a deliberately slow phase, and a bounded dependency wait. Record expected progress and heartbeat behavior for each. Use distinct job IDs and a monitor trace so one stale signal cannot explain another fixture.
Trace sender and observer
Align heartbeat writes, phase progress, worker lease or health, monitor reads, and terminal events. Identify whether the sender pauses, the observer reads late, or the worker loses ownership. Preserve an absence of records as a bounded evidence gap.
Test a phase-aware liveness signal
Exercise one heartbeat or progress update at the long-phase boundary and compare the monitor classification with the original run. Keep the blocking call’s limitation visible. Confirm that a liveness signal does not claim completion or authorize a duplicate retry.
What to carry forward
Deliver phase fixtures, aligned liveness evidence, first sender-observer divergence, and one phase-aware comparison. State whether the gap is observable silence or suspected worker loss. Define the smallest signal or monitor boundary to change and the intervention it must not trigger prematurely.
Technical background: Celery documentation.
Keep the decision with the work.
Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.
Create an account See the product workflow