Aglet

How to investigate queue backlog flow and failure isolation

A backlog investigation follows items from arrival to completion and asks why service no longer keeps pace or why one item holds unrelated work. Measure a representative queue and work class, inspect retries and dependencies, and test one bounded change while preserving ordering and durable outcomes.

Build a useful investigation brief

  1. Map arrival and service flow

    Align enqueue, visibility, claim, handler start, dependency waits, success, failure, retry, and completion for representative items. Compare arrival rate, service duration, consumer availability, queue depth, and oldest age across a healthy and backlog period. Record unmeasured transitions as unknown.

  2. Find the item or class causing pressure

    Group backlog by work class, payload size, dependency, retry count, age, and outcome. Inspect whether a poison item, lock, slow query, or downstream wait consumes capacity or blocks ordering. Keep the hypothesis provisional until item timeline and resource evidence agree.

  3. Test one bounded drain or isolation change

    Evaluate one intervention such as limiting work per item, isolating a failing class, correcting a retry path, or protecting consumer capacity for critical work. Replay the same arrival mix in a controlled environment. Check ordering, no-loss behavior, duplicate prevention, and visible completion before accepting a faster drain.

What to carry forward

Deliver an evidence-backed brief with flow timeline, pressure class, first divergence, isolation behavior, tested change, and uncertainty. Investigation is complete when implementation can target one queue boundary and verification can measure drain, ordering, and durable customer-visible outcomes.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow