Aglet

Triage Missing Fields in a Data Pipeline

A missing field can originate in the source, a parser, a join, a filter, or a final serializer. Triage should identify the affected field, records, time window, and first known boundary before anyone edits data. Capture representative rows and compare raw, transformed, and displayed values.

Establish what is happening

  1. Name the contract

    List each absent field, its expected type, and whether absence is allowed for particular record states. Pull the schema or interface contract, then mark which fields are missing consistently, intermittently, or only for one source, tenant, event type, or release.

  2. Trace the first loss

    Choose a small sample that exists upstream and follow it through ingestion, normalization, joins, storage, and presentation. Record the last stage where the value is present, the transformation responsible, and any rejection, defaulting, or null coercion observed.

  3. Set impact boundaries

    Count affected records by source, field, time window, and processing version. Compare the counts with normal traffic and downstream expectations. Separate a newly introduced gap from a long-standing contract mismatch, and preserve query text, timestamps, and sample identifiers.

What to carry forward

Output a scoped brief naming the missing fields, affected population, first observed loss boundary, and evidence quality. Stop triage when the boundary is reproducible and ownership is clear enough to queue investigation. Keep uncertainty explicit when source samples or historical comparisons are unavailable.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow