Aglet

Investigate CSV Delimiter Misidentification

A delimiter detector can choose incorrectly when punctuation inside values resembles a separator or when its sample misses a later structure change. Build a controlled file matrix that changes one format property at a time, keeping the intended columns and values explicit.

Build a useful investigation brief

  1. Reduce to a readable file matrix

    Create sanitized files with commas, tabs, and semicolons as real separators. Include the same punctuation inside quoted values and a deliberately inconsistent record. Write expected field boundaries beside each file so the parser result has an independent reference.

  2. Compare automatic and explicit parsing

    Run each file with automatic selection and with its known delimiter. Capture the chosen dialect and parsed fields, including records beyond the detection sample. If explicit parsing also fails, the evidence points beyond delimiter selection and should change the investigation scope.

  3. Trace preview settings into execution

    Follow the selected delimiter from upload through preview confirmation to the final import job. Check whether a detected value is persisted or recomputed. Compare both paths against the same file revision, rather than comparing two separately exported copies that may differ.

What to carry forward

Produce a brief naming the smallest failing file, chosen delimiter, expected fields, and the transition where settings or results diverge. Propose a change only after distinguishing detection from parsing and handoff behavior. Preserve unresolved mixed-format cases as explicit limits of the proposed fix.

Technical background: DuckDB documentation.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow