Aglet

Triage a missing heartbeat against long-job phase progress

A heartbeat can disappear during a slow but valid phase, or it can stop because the worker is gone. Triage one long-running job by aligning its liveness records with phase progress and the monitor’s threshold. Compare a short phase, a long phase, and a blocked call before escalating it as dead work.

Establish what is happening

  1. Capture the liveness timeline

    Record job ID, phase, start time, last heartbeat, last progress event, monitor observation, worker identity, and terminal state. Note the configured heartbeat and monitor intervals if available. Preserve the gap itself rather than replacing it with a dashboard’s rounded age.

  2. Compare phase-specific behavior

    Review a job that emits regular progress, one that remains in a known long phase, and one that waits on a dependency or call. Check whether the worker is still active and whether any durable progress follows the quiet period. Keep monitor classification separate from job outcome.

  3. Find the liveness handoff

    Identify the first point where heartbeat emission, phase transition, monitor read, or worker ownership stops agreeing. Compare timestamps from the sender and observer. If the worker trace ends before the monitor decision, label death versus silence unresolved.

What to carry forward

Return the job phase, heartbeat and progress timeline, monitor rule, worker evidence, and terminal outcome. Classify the scope as missing emission, slow-phase silence, observer lag, or dead ownership. Keep a liveness alarm provisional when no independent worker or durable state confirms it.

Technical background: Celery documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow