Aglet

Investigate long-job timeouts around the commit boundary

Investigation should make the deadline’s behavior concrete at the point where durable work changes. Use a fixed input and hold the job near a known commit, cleanup, and dependency boundary. Capture the signal, worker exit, records, and retry decision in order so the brief explains what is safe to repeat.

Build a useful investigation brief

  1. Prepare deadline fixtures

    Create sanitized jobs that finish below the limit, pause immediately before a commit, and pause after the commit while cleanup remains. Add a dependency-wait case if it uses the same budget. Record expected effects and terminal states before running the fixtures.

  2. Align signal with durable records

    Trace deadline calculation, limit signal, current phase, transaction or commit events, cleanup, worker exit, and retry creation. Compare their timestamps and identifiers. If logs disagree with durable state, make the reconciliation gap part of the brief instead of choosing the convenient record.

  3. Test one deadline margin

    Exercise one bounded margin or phase-aware cleanup rule against the same fixtures. Compare partial effects, terminal wording, and retry eligibility with the original. Keep any dependency or worker variability separate so a changed margin is not credited for an unrelated timing shift.

What to carry forward

Deliver boundary fixtures, ordered timeout and commit evidence, cleanup result, and one tested margin or phase rule. State the smallest safe change and its retry assumption. Preserve uncertainty around external effects or missing clocks rather than declaring a timeout cause from elapsed duration alone.

Technical background: Celery documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow