Background jobs playbooks
Keep long-running work understandable through progress, scheduling, fairness, and recovery. Choose a scenario, then the decision you need to make.
Batch progress stays blank or misleading
A batch runs, but its progress view cannot distinguish queued, active, completed, failed, or excluded work.
Batch cancellation takes too long to take effect
A requested batch cancellation does not clearly stop queued work, active steps, or later side effects.
Batch resume starts at the wrong checkpoint
An interrupted batch restarts from the beginning or advances past items that were not durably completed.
Batch completion hides partial item failures
A batch summary reports completion without showing that individual items failed, were skipped, or need review.
Batch work overruns its operating window
A batch continues past the period in which it is allowed to run and collides with the next operational boundary.
Job priority is inverted at dispatch
A job marked urgent waits while lower-priority work is selected first, especially when queues or worker classes differ.
Low priority jobs are starved
Lower-priority work remains pending while a steady stream of favored jobs consumes every available worker.
Queue service fairness drifts between work classes
A queue that once shared workers across tenants or work classes gradually lets one class consume most service.
Worker concurrency reduces useful throughput
Adding worker threads or processes increases contention and makes total work slower or less predictable.
Long running job loses its heartbeat
A long task appears dead because its liveness signal stops during a slow phase even though work may still be progressing.
Long running job hits the wrong time limit
A job reaches a soft or hard time limit that does not match its cleanup boundary, leaving work or state unclear.
Scheduled job shifts across time zones
A recurring job runs at an unexpected local time when its configured zone, calendar rule, or offset changes.
- Triage scheduled timezone drift from local rule to UTC run
- Prioritize scheduled timezone drift by missed calendar consequence
- Investigate scheduled timezone drift across a calendar boundary
- Verify scheduled runs before and after a timezone transition
- Keep an explicit timezone contract for recurring jobs
Scheduled run misses its due window
A recurring run becomes due but never appears in the expected execution path or becomes visible too late to recover cleanly.
Scheduled runs overlap unexpectedly
A recurring schedule starts a new run before the previous occurrence has finished or reached its terminal state.
Job output expires before it is read
A completed job has a result that disappears before the person or downstream step can use it.
Cancellation arrives after a job commits
A cancellation request races with a durable commit and the visible status does not explain which outcome won.
Ready jobs wait too long for a claim
A job is ready for work but spends most of its lifetime waiting before a worker claims or starts it.
Worker shutdown interrupts active jobs
Stopping a worker leaves active jobs with unclear checkpoints, terminal state, or recovery ownership.
Job dependency waits on the wrong prerequisite
A dependent job remains blocked or starts too early because prerequisite state is stale, incomplete, or mapped incorrectly.
Batch size change causes a regression
Changing the number of records handled per batch shifts memory, runtime, query behavior, or completion correctness.
- Triage batch-size regression with matched records and resources
- Prioritize batch-size regression by correctness and resource risk
- Investigate batch-size changes with a bounded size sweep
- Verify batch sizes for coverage, runtime, and resource headroom
- Document a tested batch-size range and its resource guard