Skip to content

fix: detect CSV delimiter by consistency, not first-occurrence - #5166

Open
krMaynard wants to merge 3 commits into
ankitects:mainfrom
krMaynard:submit/csv-delimiter
Open

fix: detect CSV delimiter by consistency, not first-occurrence#5166
krMaynard wants to merge 3 commits into
ankitects:mainfrom
krMaynard:submit/csv-delimiter

Conversation

@krMaynard

@krMaynard krMaynard commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Linked issue (required)

Fixes #5253.

Also refs #3853, which reported the same symptom and is closed. That report had two causes: the mangled #separator:Comma,, header line, fixed at the time by trimming trailing delimiters from comment lines, and the underlying auto-detection favouring : over , whenever both appear, per @dae's analysis there. This PR fixes the second, which is still reachable today on any file without a valid #separator: header.

Summary / motivation (required)

delimiter_from_reader returned the first delimiter byte found anywhere in the 8KB sample, in a fixed enum order that tries : before ,. So a normal comma CSV containing a colon in any field (a time like 9:00, a URL, a parent::child tag, "Note:") was detected as colon-delimited and split into the wrong columns, corrupting the import. This is the content-driven half of the misdetection discussed in #3853; the header-line half was already fixed by trimming trailing delimiters from #-comment lines.

The fix keeps detection dependency-free: parse the sample with each candidate delimiter and pick the one that produces the most consistent positive field count. Because each candidate is parsed as CSV, delimiter-like characters inside quoted fields do not affect the score. Ties use a priority order that keeps genuine delimiters ahead of content-prone ones such as colon and space. For example, a semicolon file with decimal commas (1,5;2,7) stays semicolon.

Steps to reproduce (required, use N/A if not applicable)

  1. Save as test.csv (no #separator: header):
    time,note
    9:00,wake up
    10:30,run
    
  2. Import via File → Import.
  3. The preview splits rows on : instead of ,.

How to test (required)

Checklist (minimum)

  • I ran ./ninja check or an equivalent relevant check locally.
  • I added or updated tests when the change is non-trivial or behavior changed.

Details

  • New unit tests in metadata.rs cover comma files with colons in fields, quoted fields containing another candidate delimiter, semicolon files with decimal commas, and genuinely colon-delimited files. The existing detection cases still pass.
  • Upstream CI for the latest commit: https://github.com/ankitects/anki/actions/runs/31341127603
  • The new regression test and all 539 Rust tests passed. Format and minilints also passed. The workflow is red because the current Python suite reports 63% coverage against the repository's 65% floor; this PR does not touch Python.

Before / after behavior (optional)

Before: first-occurrence detection mis-splits comma CSVs containing colons (or any delimiter byte appearing earlier in the enum order than the real one). After: consistency-based detection reads them correctly; explicitly-specified delimiters are unaffected.

Risk / compatibility / migration (optional)

Only the auto-detection fallback changes; #separator: headers and delimiters passed explicitly through the import dialog are untouched. Detection remains a heuristic and can still guess wrong on genuinely ambiguous files. The tie-break deliberately prefers comma and semicolon over colon and space, so a colon-delimited file whose every line also contains exactly one comma now resolves to comma where the old code resolved to colon. That trade-off favors the far more common case (comma/semicolon files with colons in content) at the expense of a rare one, and any file with a consistent delimiter and no equally-consistent impostor is detected correctly.

UI evidence (required for visual changes; otherwise N/A)

N/A

Scope

  • This PR is focused on one change (no unrelated edits).

delimiter_from_reader returned the first delimiter byte found anywhere in
the sample, in a fixed enum order that tried ':' before ','. So a normal
comma CSV containing a colon in any field (a time like 9:00, a URL, "Note:")
was detected as colon-delimited and split into the wrong columns, corrupting
the import.

Instead, pick the delimiter that splits the first several lines most
consistently (the same positive count on the most lines), which ignores a
delimiter that only appears inside field content. Break ties with a priority
order that keeps genuine delimiters ahead of content-prone ones (colon,
space), so an ambiguous file is read the way it was most likely written -
e.g. a semicolon file with decimal commas (1,5;2,7) stays semicolon.
@krMaynard
krMaynard marked this pull request as ready for review July 16, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV import: delimiter auto-detection returns the first delimiter byte found, not the one that splits the file

1 participant