Aglet

Define phase-aware heartbeat expectations for long jobs

The lasting lesson is that liveness belongs to a job phase and an owner, not to one timestamp interpreted without context. Document what a heartbeat proves, what phase progress proves, and how a monitor handles silence. Keep the recovery decision separate from the eventual success or failure of the job.

Keep the lesson for the next incident

  1. Write the liveness contract

    Name the heartbeat source, phase or lease it covers, monitor interpretation, and action after the signal ages out. State which phases may be quiet and what independent record distinguishes blocked, active, and abandoned work.

  2. Retain phase fixtures

    Keep short, long, and dependency-wait jobs with expected heartbeat, progress, monitor, and terminal behavior. Re-run them after adding a long phase, changing monitor intervals, altering leases, or moving heartbeat emission to another worker boundary.

  3. Watch recurrence by phase

    Review aged heartbeats with no phase progress, workers that retain ownership without a signal, and monitors that classify a known wait as dead. Inspect sender and observer timestamps before tuning thresholds. Assign review when a new phase lacks a liveness expectation.

What to carry forward

Preserve a phase-aware liveness contract, independent ownership evidence, and short, long, and blocked fixtures. Recurrence checks should connect heartbeat age to phase progress and monitor action. Reopen review when a new long step, lease, observer, or recovery path changes what silence means.

Technical background: Celery documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow